https://linen.dev logo
Join Slack
Powered by
meltano run does not seem to pass the state define...
# troubleshooting
d
meltano run does not seem to pass the state defined in the meltano.yml file to taps, is this a known issue ? I’ve checked the logs in debug mode and found the line where it is invoking the tap and target. I see that config as well as other information is passed as command line arguments, however the state file that I’ve defined in the yaml file is not passed.
p
I dont know if thats an expected feature or not (@florian.hines might know) but one thing I think you can do is use the 
state
command to set your state prior to running https://docs.meltano.com/reference/command-line-interface#set using the 
--input-file
flag
For example I think this would work 
meltano state set --force dev:tap-gitlab-to-target-jsonl --input-file gitlab_state.json
then 
meltano --environment=dev run tap-gitlab target-jsonl
d
I will give this a try, I suppose I’m just unsure how meltano actually shares any variant of the state with the tap. Right now it only seems to grab the output after the tap completes and saves it in the DB
Should I not be expecting this not bi-directional states
p
No your expectations are correct, it should store your state and retrieve it before the next run and pass it to the tap. Maybe something to note is https://docs.meltano.com/reference/command-line-interface#using-run-with-environments 
Note that if no environment is active, meltano run _does not_ generate a state ID and it does not track state.
Are you using environments?
d
yes just ran with an environment no dice
particularly i’m looking at the Invoking debug log and I don’t see the flag being passed there either
Copy code
Invoking: ['/opt/meltano-pipelines/.meltano/extractors/tap-intercom/venv/bin/tap-intercom', '--config', '/opt/meltano-pipelines/.meltano/run/tap-intercom--contacts/tap.6c767406-6b60-4d28-a710-55ed3881b2fe.config.json', '--catalog', '/opt/meltano-pipelines/.meltano/run/tap-intercom--contacts/tap.properties.json']
p
ahhh I think I know whats going on. The tap doesnt advertise that it supports state https://hub.meltano.com/extractors/tap-intercom, so meltano thinks it cant support it. I think thats a bug on the hub because https://github.com/singer-io/tap-intercom looks to support state
You can add 
state
as a capabilities list in your meltano.yml to see if that fixes it
Copy code
pip_url: xyz
capabilities:
    - catalog
    - state
    - discover
Whether that solves your problem or not I created a PR to the hub to fix that problem https://github.com/meltano/hub/pull/688 😄
d
Wow
Adding state to capabilities seemed to fix it
At least with defining a state
Should I expect for the state to be passed automatically from the system if I do not specify a state file in the config
p
Awesome!! Yes meltano will manage state for you behind the scenes
Feel free to open an issue related to this if you have any feedback on how we can make this clearer for anyone else dealing with similar behavior in the future
Maybe we can print a log message that state isnt supported for that particular tap so users are aware
d
Thank you for the assistance. Just finished testing without the state properties in meltano.yml I do see that the --state flag is passed when meltano invokes the plugin. Having the capabilities is definitely a requirement here. I also see that the tap is able to retrieve the right updated_at bookmark
Scouring through the docs for a whole day not sure how I missed that requirement , or if its undocumented
p
No that was a bug with the tap's plugin definition, not on you 😅
That update got merged https://github.com/meltano/hub/pull/688 so I think if you run
Copy code
meltano lock tap-intercom --update
it should pull in that change into your lock file so you dont need to manually add it to meltano.yml anymore
2 Views
Open in Slack
PreviousNext
Powered by