Hello, I’m looking to deploy a new version of meltano using the new environment configuration. How can I properly link the system db configurations to the new “prod” environment to ensure that incremental processes pick up where they left off as recorded in the db? Also, am I understanding incremental processes correctly where the previous jobs are stored in the system db as opposed to the last upload in the DW?

Hi @drew_ipson, the environments feature is agnostic to the system db so you shouldn't need to change anything on that front, just continue using the same

--job_id

Also, am I understanding incremental processes correctly where the previous jobs are stored in the system db as opposed to the last upload in the DW?

That's correct. State is only stored in and retrieved from the system db.

@edgar_ramirez_mondragon thank you for the clarification! I've been running tests with new taps in a development environment that loads the data to a dev db. I am now ready to begin loading data into the production db but would like to continue extractions from the last point of production load. How would I handle this in meltano since the state records are agnostic to environments?

@drew_ipson are you saying you have a job_id that already existed in your production environment that youre trying to use with your new tap? In that case you can just run your new tap with the same job_id as before and the old state should get picked up.

@pat_nadolny yes, I'd recommend using different job_ids to track the state of sync in different environments otherwise you will not get all of it in prod. @drew_ipson If the dev db and prod db are different instances I'd recommend starting fresh in prod with a new job_id

@pat_nadolny Thank you for your input on this. I am not aware of specifying a job id for taps as I thought they were generated. Can I just pass the job_id value I want via the flag in the CLI?

@drew_ipson yes just include it in the CLI command. Check out https://meltano.com/docs/getting-started.html#run-a-data-integration-el-pipeline and https://meltano.com/docs/integration.html#incremental-replication-state for more details. I'm actually just adding some more call outs around the importance of

job_id

in the docs