Hi @connor_lough - yes, that is the idea, but a few caveats: 1. You need an active environment https://docs.meltano.com/reference/command-line-interface#run a. Note that if no environment is active,

meltano run

does not generate a State ID and it does not track state. 2. The tap must support it. We recently proved that the tap-mssql we forked does this correctly: https://github.com/Matatika/pipelinewise-tap-mssql a. There are several variants, check yours b. date-time https://github.com/Matatika/pipelinewise-tap-mssql/blob/master/tap_mssql/sync_strategies/incremental.py#L58

Hey Aaron, what are you using for orchestration? I've got Airflow firing up a Docker image with K8s and now I'm realizing that there isn't a way to keep track of the updated 'state'. What would you recommend? Writing the state to S3 on Airflow job finish, and replace the file on new job startup?

Hi Connor, we are using Spring Cloud Dataflow - which launches a pod on K8s just like Airflow. We set the MELTANO_DATABASE_URI to a postgres database we created for each of our client workspaces. Meltano is persisting this updated state throughout the run. Doing a state move at the end seems like a bad idea - what if the job never finishes? The separate DB for state design is ok, especially when we consider meltano might be moving data to many targets - some of which might not be transactional. I spent some time thinking about this before we jumped in and committed to using meltano. For integration use cases, I'd ideally like the state and the rows to be in the same target database, in the same transaction, so neither rows nor state can be committed without the other. Meltano writes the state 2nd, so you'll always get a row at least once. This is fine for data import use cases.

Aaron, again, thank you! So where I was thinking of writing the state.json file to S3, you have the same information being put into a postgres database? Now that you've been using that method for a while (I assume), lessoned learned? Different approaches? I'm going to link our thread in the best-practices channel at large to see if other folks have thoughts.

Right. Setting the meltano system database to an external Postgres is the recommended approach: https://docs.meltano.com/guide/production#storing-metadata Couple of years of ‘meltano elt…’ in our platform without issues. ‘meltano run’ required us to set a dummy environment, but that's a transitional requirement from elt where we set the state id https://docs.meltano.com/reference/command-line-interface#run State feels like one of the crucial things to understand when moving from poc to prod, but so far it does just work.

Thanks for the useful discussions! I'm currently evaluating Meltano as part of a poc but started thinking how I could productionize it and "state" seems to be one of these things which are core. We already use Airflow Cloud Composer and I would've thought there would be a way to pass state from Airflow, since it already has a metadata database but I don't think that is possible.

Ping me if you'd like a chat about it. Happy to jump on a call and learn a bit about how you are setting things up and vice versa