https://linen.dev logo
Join Slack
Powered by
Hi team, I have an issue with creating a custom ta...
# announcements
b
Hi team, I have an issue with creating a custom tap. When I run 
meltano invoke <my_tap>
, it does output the correct config. But when running 
meltano invoke <my_tap>
, it fails. I see the command passed is 
command ['/<my_project>/.meltano/extractors/<my_tap>/venv/bin/<my_tap>', '--config', '/<my_project>/.meltano/run/<my_tap>/tap.config.json', '--discover']
However, when checking 
/<my_project>/.meltano/run/<my_tap>/tap.config.json
it looks like this file is not created. Is it expected? Is the file created and then removed completely? thanks!
d
Hi @benjamin_maquet! Let's get to the bottom of this šŸ™‚
When I runĀ 
meltano invoke <my_tap>
, it does output the correct config.
I assume you mean 
meltano config <tap>
?
But when runningĀ 
meltano invoke <my_tap>
, it fails.
How does it fail? Can you share any relevant error output?
I see the command passed isĀ 
command ['/<my_project>/.meltano/extractors/<my_tap>/venv/bin/<my_tap>', '--config', '/<my_project>/.meltano/run/<my_tap>/tap.config.json', '--discover']
However, when checkingĀ 
/<my_project>/.meltano/run/<my_tap>/tap.config.json
Ā it looks like this file is not created. Is it expected? Is the file created and then removed completely?
Yep, the config file is deleted automatically to prevent sensitive values (credentials etc) from sticking around on your filesystem. If you're running 
meltano invoke
with 
--log-level=debug
, you should see a log message confirming the deletion as well.
You can run 
meltano invoke --dump=config <tap>
to print the contents of the generated 
tap.config.json
instead of actually invoking the plugin: https://meltano.com/docs/command-line-interface.html#invoke
meltano invoke --dump=config
typically shows the same result as 
meltano config
, but it lets you print the actual generated config file
b
Hi @douwe_maan, thanks a lot for the quick answer. It fails with invalid credentials, but I'm sure these creds work. I thought maybe the config.json file was not created but I guess it is because 
meltano config <tap>
seems to work correctly.
I'll continue to investigate what is happening, there seems to be a problem with credentials I provide in the 
meltano.yml
file
d
@benjamin_maquet Since 
meltano invoke --dump=config <tap>
shows the content of 
tap.config.json
exactly as it'll be passed to the plugin, if that file looks correct but you're still getting a creds error, the issue is likely in the tap itself, or the config keys used may be incorrect.
Let me know if you'd like to jump on a quick call so I can help you get to the bottom of this šŸ™‚
b
thanks @douwe_maan! I'll try to fix this and let you know if I need some support again. I think this is on my end or the API end...!
And actually, 
meltano invoke --dump=config <tap>
does not work since it's failing before showing the content of the file
d
Hmm, what's it failing on?
If it's printing an error but no stacktrace, 
meltano --log-level=debug invoke --dump=config <tap>
may show more
I think I know why 
meltano invoke --dump=config
is failing: before invoking the tap in sync mode, Meltano always runs the tap in discovery mode to generate the catalog file, and using 
--dump=config
doesn't disable that "before_invoke" hook, so what you're seeing is the tap failing in discovery mode before Meltano even gets a chance to print the generated config file.
b
yes I agree. I see 
--discover
in the command passed
d
I've created an issue: https://gitlab.com/meltano/meltano/-/issues/2461 Even if 
meltano invoke --dump=catalog
isn't working, we should still be able to trust that whatever 
meltano config
prints is what actually gets passed to the plugin as 
config.json
. Were you able to figure out why it's complaining about your credentials?
6 Views
Open in Slack
PreviousNext
Powered by