How does Meltano handle state files? I've had trouble with Singer state files in the past, and am curious about how Meltano works with state files. To be more specific, if I have a scheduled pipeline, how does it determine where to pick up?

https://meltano.com/docs/integration.html#incremental-replication-state

When meltano elt is run a subsequent time, it will look for the most recent completed (successful or failed) pipeline run with the same job ID that generated some state. If found, this state is then passed along to the extractor.

For scheduled pipelines, the schedule name is used as the job ID

I have a followup question to this. What happens if you use a

--select

in a run to backfill a part of the job. For example: •

job_id=foo

is selecting a

users

table using Binlog replication. It’s been run on a schedule for a while and is up to date • I want to add a HUGE table. Let’s call it

results

. • Outside of the job I run

meltano elt … job_id=foo --select results

. _I want the same job_id because after the backfill they’ll be part of the same airflow task_ • The next time I run will it do a full-refresh on

users

since it wasn’t part of the pervious run and can’t find a

binlog

?

The next time I run will it do a full-refresh on 

users

 since it wasn’t part of the pervious run and can’t find a 

binlog

?

@matt_cooley No, it should incrementally replicate both

users

and

results

as expected, since runs with

--select

or

--except

are stored as incomplete, and their state is merged into the most recent complete run's state to create the state for the next incremental run, so the next run should see bookmarks both for

users

and

results

Thanks! Sounds good. It’s mostly worked that way. There’s one instance where it appeared not to, but I don’t have good stats on it. I’ll make note of what I see if it happens again.

OK, please do!