Andy Carter
06/28/2023, 9:05 AMstate
to decide where to start querying a tap via timestamp filter, but NOT write the updated state at the end of a meltano run
back to the state store?
Reasoning: I use a Azure container for state storage in test/prod. In local dev, I sometimes want to point to the existing state marker in test/prod as a starting point to reference new data, rather than a local run that brings in tons of data. But I don't want to update the state as that would interrupt my test/prod processes. That way I can iterate on the same small dataset each time. Basically I would like the option to treat the state as immutable/read only.visch
06/28/2023, 12:31 PMstart_date
I'd try to use that as it's easier than manipulating state directly and it's made for what you're trying to do right now. If you're really set on using state
directly then maybe something like meltano state get
to get your production state, pipe that to a file, then use meltano state set
to store the state from production to your dev local side.Andy Carter
06/28/2023, 1:24 PM