https://linen.dev logo
Join Slack
Powered by
How can I delete plugin?
# announcements
m
How can I delete plugin?
d
@meir.shamay There's not currently a 
meltano remove <plugin>
command, but you can directly remove plugins from your project by simply modifying 
meltano.yml
m
thanks
I am getting an error that remove action is not supported
d
Right, like I mentioned, there is not currently a 
remove
command 😉 I suggest editing 
meltano.yml
directly
m
I want to reinstall plugin, and I saw that melano install, not doing the updates, so, I thought, remove it and add it again
d
@meir.shamay Ah okay! 
meltano install
does do updates as long as the PyPI package version number is different from the currently installed version. If that's not behaving as expected, you can manually delete the plugin's installation directory at 
.meltano/<plugin type>/<plugin name>
and rerun 
meltano install
m
ok 👍
I have an issue that the json files, like config.json, state.json can't be loaded when I am running using meltano.
I think that the install is not taking the json files from the package directory.
d
@meir.shamay Are you seeing any errors? Note that you're not expected to manage your own 
config.json
and 
state.json
as Metlano handles this for you: https://meltano.com/#meltano-config
m
I am sorry for the ongoing questions, but, I have this scenario, I need to pass an arguments (email address, account id) to the tap. The tap itself needs to read credentials from local file. So, when I am trying to read the credentials.json file I am getting an error that it's not found, although, it's part of the package, and the other issue is, passing the arguments to the tap.
d
@meir.shamay No worries about the questions, answering them is my job 😄 What tap are you using? Can you share the error output?
@meir.shamay For passing those other arguments, I'd suggest using 
meltano config
to store them in 
meltano.yml
, or passing them as environment variables at invocation time
m
I developed an extractor tap that reads data from gmail, the tap is working when I am running it using command line, but, the tap itself is using credentials.json file to access google api
and when I am installing it into meltano, I am getting an error that this file is not exists
this is the error:
No state was found, complete import.
before_invoke hook 'run_discovery_hook' has failed: Schema discovery failed: command ['/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/bin/tap-gmail', '--config', '/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/run/tap-gmail/tap.config.json', '--discover'] returned 1: CRITICAL [Errno 2] No such file or directory: 'config.json' Traceback (most recent call last): File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/bin/tap-gmail", line 11, in <module> load_entry_point('tap-gmail', 'console_scripts', 'tap-gmail')() File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/singer/utils.py", line 229, in wrapped return fnc(*args, **kwargs) File "/Users/may/development/project/tap-gmail/tap_gmail/__init__.py", line 42, in main gmail_client = client_init(parsed_args.config) File "/Users/may/development/project/tap-gmail/tap_gmail/__init__.py", line 30, in client_init credentials = service_account.Credentials.from_service_account_file('config.json', scopes=SCOPES) File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 226, in from_service_account_file filename, require=["client_email", "token_uri"] File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/google/auth/_service_account_info.py", line 72, in from_filename with io.open(filename, "r", encoding="utf-8") as json_file: FileNotFoundError: [Errno 2] No such file or directory: 'config.json' CRITICAL [Errno 2] No such file or directory: 'config.json' Traceback (most recent call last): File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/bin/tap-gmail", line 11, in <module> load_entry_point('tap-gmail', 'console_scripts', 'tap-gmail')() File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/singer/utils.py", line 229, in wrapped return fnc(*args, **kwargs) File "/Users/may/development/project/tap-gmail/tap_gmail/__init__.py", line 42, in main gmail_client = client_init(parsed_args.config) File "/Users/may/development/project/tap-gmail/tap_gmail/__init__.py", line 30, in client_init credentials = service_account.Credentials.from_service_account_file('config.json', scopes=SCOPES) File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 226, in from_service_account_file filename, require=["client_email", "token_uri"] File "/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/lib/python3.7/site-packages/google/auth/_service_account_info.py", line 72, in from_filename with io.open(filename, "r", encoding="utf-8") as json_file: FileNotFoundError: [Errno 2] No such file or directory: 'config.json' ELT could not complete, an error happened during the process: Subprocesses didn't exit cleanly: tap(1), target(0) Traceback (most recent call last): File "/Users/may/development/project/montara/meltano/.venv/lib/python3.7/site-packages/meltano/cli/elt.py", line 73, in elt run_extract_load(elt_context, session, dry_run=dry) File "/Users/may/development/project/montara/meltano/.venv/lib/python3.7/site-packages/meltano/cli/elt.py", line 104, in run_extract_load singer_runner.run(session, **kwargs) File "/Users/may/development/project/montara/meltano/.venv/lib/python3.7/site-packages/meltano/core/runner/singer.py", line 158, in run loop.run_until_complete(self.invoke(tap, t…
I changed the file name to config.json just to check
d
@meir.shamay Is the tap working correctly when the executable is invoked directly instead of through Meltano? I wonder if 
credentials.json
is not getting packaged up correctly (did you reference it from 
package_data
in 
setup.py
?) or if it's not read from the correct path
Are you actually storing creds in the 
credentials.json
file? You may not actually want to check those secrets into the package repo 🙂
m
I refer to inside the setup.py
yes, I know, thanks, it's ignored in the .gitignore for now
d
What if you move 
credentials.json
to your project root instead of the package root? Since the project root is always the current working directory when invoking taps and targets, it may be looking there
m
ok, checking
no, same issue
d
Same error too? What about putting a breakpoint there and seeing if you can figure out the full path that is supposedly not found?
To see in which directory it's actually looking
m
it's looking inside the current run directory
d
@meir.shamay Ah, interesting. It's not currently possible to put files there manually. Would it be an option to use the regular config options to configure your credentials, and then create the appropriate 
credentials.json
on the fly in your tap based on those?
Managing everything through settings will give you a better Meltano experience 😉
m
ok
d
m
ok, now it's completed
but, I don't see the data
d
What target did you use? Does the 
meltano elt
output say anything about the target?
m
target-jsonl
d
@meir.shamay Are you using the default 
output
destination directory?
You can run 
meltano config target-jsonl
to see how it's currently configured
m
{"destination_path": "output", "do_timestamp_file": false}
d
All right. But there's nothing in your output directory? Do the logs indicate any streams actually being output by your tap? Consider running “meltano invoke tap-yourtap” to verify that, you’ll see all the Singer messages printed right to your stdout
m
this is how that data passed, using the stdout?
d
The tap and target communicate over a pipe, yes! Meltano basically just runs 
tap | target
m
ok, I will check that the data is printed out
why I would like to use airflow instead of meltango schedule?
is it about scaling?
d
“Meltano schedule” only stores the schedule in 
meltano.yml
, but it doesn’t actually handle the execution, since that will depend on your environment. Airflow is one tool you could use (and we provide a handy default DAG generator to parse the meltano.yml schedules and turn them into Airflow DAGs), but you could also use cron or any other orchestration tool
You don’t need to use “meltano schedule” at all if you’re fine setting up some other mechanism to run “meltano elt ...” when appropriate
m
👍
Hi again, I am checking the tap using 
meltano invoke tap-gmail
but, nothing is print out
although I am writing the records
and I see them when I execute it without meltano
I guess this is the reason why the "Explore" button is gray
d
@meir.shamay Hmm, so you are seeing output when you run 
.meltano/extractors/tap-gmail/venv/bin/tap-gmail
directly and pass the correct configuration flags manually?
m
yes
d
Can you share the exact flags you’re passing when you run it like that?
m
python init.py --config config.json --catalog catalog.json --state state.json
d
Ah, it’s possible that the local version works, but not once it’s been installed into the virtual environment. Did you specifically run the version in that long directory path I shared, which is where “meltano install” will have installed your tap?
m
I can try that, anyway when I run this: 
meltano elt tap-gmail target-jsonl
this is the output
Running extract & load... No state was found, complete import. Extract & load complete! Transformation skipped.
d
Hmm. If the tap’s executable were failing with an error code, that command would as well, so that’s not happening and tap is running succesfully. We’d definitely expect to see more output though.
m
when I run inside the path you mentioned
./tap-gmail --config config.json --catalog catalog.json --state state.json
I am getting an error: No such file or directory: 'config.json'
d
Can you point it at the versions of those JSON files you were using when invoking 
___init___.py
directly?
m
you mean the full path?
d
Yeah, otherwise it’ll look locally where the files don’t exist
m
now it's working, and I see the output records
d
Interesting! But 
meltano invoke tap-gmail
wasn’t giving anything? Can you run it when 
meltano --log-level=debug invoke tap-gmail
?
m
yes
d
Meltano should make it much easier to debug these kinds of issues, for what it’s worth, but for now I’ll gladly help you
m
ok, I see the errors, it looking for the json files
d
Ah, can you share the errors or full output of that command?
m
it contain some sensitive info, the env data
this is whithout the env data
[2020-07-14 16:20:05,772] [14579|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x10e602d10>@sqlite:////Users/may/development/project/montara/meltano/montara_pipelines/.meltano/meltano.db
[2020-07-14 162005,955] [14579|MainThread|root] [DEBUG] Created configuration stub at <_io.TextIOWrapper name='/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/tap.config.json' mode='w' encoding='UTF-8'> [2020-07-14 162005,957] [14579|MainThread|root] [DEBUG] Could not find tap.properties.json in /Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/tap.properties.json, skipping. [2020-07-14 162005,957] [14579|MainThread|root] [DEBUG] Could not find state.json in /Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/state.json, skipping. [2020-07-14 162006,032] [14579|MainThread|root] [DEBUG] Invoking: ['/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/extractors/tap-gmail/venv/bin/tap-gmail', '--config', '/Users/may/development/project/montara/meltano/montara_pipelines/.meltano/run/tap-gmail/tap.config.json', '--discover']
d
All right, so judging by the last line, it’s running discovery mode to generate a catalog file. Does your tap implement discovery mode? Does running that invoked command correctly generate a catalog file as expected?
(You’ll want to join the array elements by spaces to so that each element becomes an argument, if you want to run that command in your shell)
m
I already have the catalog file inside the tap, it's not going to change, I only want to run the sync
d
Ah okay! Then I suggest removing the “discover” capability from your tap’s custom definition in meltano.yml
Otherwise it’ll try to run discovery mode first, unnecessarily
m
ok, I will do that
d
If you’re bundling the catalog file with your tap, you’ll also want to modify the code not look at the “catalog” flag, but just open the packaged file directly
m
ok, I will do the changes, about the config file it's looking, I can pass it as env variable, the path to the file?
d
The main “config.json” for your tap will be generated automatically based on your tap’s settings and the values set in meltano.yml or the environment (see “meltano config tap-gmail list”), which is what I’d recommend you use for all config if possible, but if you want to pass an additional path to a separate config file you can do so using an env var if you like
m
ok, I will want to run it using airflow, but, when I run the airflow ui, I don't see the DAG I created inside meltano.
d
@meir.shamay Do you have the 
orchestrate/dags/meltano.py
file in your project? And you've used 
meltano schedule
to schedule the pipelines, and are running Airflow using 
meltano invoke airflow scheduler
and 
meltano invoke airflow webserver?
m
I will do that. You know, I wonder if you have like docker-compose file that demonstrate a production environment, with all the features that you need to scale, I think it will help a lot of people to see it in one place and learn from it.
also, I would add integration with gitlab ci/cd feature to demonstrate deployment to new changes to taps
d
@meir.shamay Thanks, that's useful feedback. Did you see https://meltano.com/#containerization and https://meltano.com/docs/production.html? Reading those and following all of the example code on the homepage should hopefully be enough to get someone started with existing taps and targets (even if they need to be added as custom plugins), but your situation is definitely a little tricker since you're building your own tap at the same time. A user is also currently working on Helm charts to allow easy deployment onto Kubernetes: https://gitlab.com/meltano/meltano/-/issues/2149 and there's this issue to provide a better docker-compose.yml example: https://gitlab.com/meltano/meltano/-/issues/2106, but it would initially be focused on setting up a development environment, not a production environment, because Docker Compose by itself is not a recommended production deployment strategy
@meir.shamay The goal is to in the future also build tooling into Meltano to make it easier to actually develop and test and maintain new taps, instead of just using existing ones, but we're not there yet.
In general, though, I really appreciate the feedback and if you have specific ideas or suggestions or run into bugs, I'd love it if you could create issues or comment on existing ones since that's how we keep track of all of the remaining fork, to be picked up by me or the community 🙂
m
yes, sure, you are great, thanks.
d
Thanks a lot!
8 Views
Open in Slack
PreviousNext
Powered by