Hello everyone,
does meltano store all the meta in...
# troubleshootingm
mari_n
03/18/2022, 8:10 AMHello everyone,
does meltano store all the meta information regarding the extractions and the updated at stuff? If so can we manipulate it?
t
thomas_briggs
03/18/2022, 1:10 PMI need to know this too. I have a single table I need to reload and I think tweaking the state data is probably the only way. 🤨
p
pat_nadolny
03/22/2022, 3:16 PMI believe right now you'll have to manually update the system database state record in the jobs table but theres an open issue that was discussed in the last office hours around creating a
meltano statepat_nadolny
03/22/2022, 3:16 PMIf you have thoughts definitely add a comment on that issue!
t
thomas_briggs
03/22/2022, 3:47 PMThanks @pat_nadolny. In my test env I've been able to do it by dumping state with the --dump option, tweaking that file, and then running again with the --state option. That only works with
meltano eltmeltano runthomas_briggs
03/22/2022, 3:47 PMThe new state options would be a way better option though
p
pat_nadolny
03/22/2022, 3:50 PM@thomas_briggs yeah that sounds like a reasonable work around too. I actually think incremental state is supported in
runt
thomas_briggs
03/22/2022, 4:36 PM@pat_nadolny agreed that it's generally supported. (Which definitely made my life better, BTW!) I don't see a way to override the state for a single tap/pipeline though, which is what the --state option to the elt command does (if only because elt only operates on one pipeline...)
thomas_briggs
03/22/2022, 4:38 PMIt feels to me like a similar option will ultimately be necessary for the run command though. For example I have a pipeline that's mysql -> meltano-map-transformer -> snowflake. If I need to reload a single table I can't just take the transformer out of the pipeline in order to enable overriding the state. Easy for me to say since I don't have to implement it though. 😝
p
pat_nadolny
03/22/2022, 4:54 PM@thomas_briggs ahh ok I understand what youre saying now. Good point. I would encourage you to add that context to the issue I linked in case that hasnt already been discussed
t
thomas_briggs
03/22/2022, 8:06 PMDone