How does Meltano behave when a run is started before the previous has completed? I can't find that in the docs. Example where this could happen is with a scheduler running hourly and a run that takes over an hour

this would just be an airflow thing right? If so does https://stackoverflow.com/a/57229106/2291321 answer that for you?

Thanks Tom, sounds like we should definitely enable

depends_on_past

in the dag generator then?

@niall_woodward

meltano elt

could check the system database to see if a job with the same job ID is already running, but it doesn't currently, so 2 parallel

meltano elt

s would start with the same incremental state (of the most recent complete run) and would both try to extract (and load) all the new data. In that scenario, it'd be up to the loader and the database to use UPSERT, primary keys, and uniqueness constraints to prevent duplicate records from being created. If you think

meltano elt

should behave differently here, can you please create an issue on https://gitlab.com/meltano/meltano/-/issues? Like Tom mentioned, you should be able to configure your orchestrator to not start a new run if the one for the previous interval tick hasn't finished yet. If the appropriate option for Airflow is

depends_on_past=True

or

max_active_runs=1

, we should add that to https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py as well: https://gitlab.com/meltano/files-airflow/-/issues/2