https://meltano.com/ logo
#cli
Title
# cli
a

aloof-twilight-67541

10/29/2020, 4:35 PM
For an ELT pipeline, is it a problem if two jobs (with the same job id) overlap with each other. For example let's say the full ELT takes ~30 minutes, but the previous ELT took 40 minutes. (cron job is set to run every 30 minutes)
1
r

ripe-musician-59933

10/29/2020, 4:45 PM
@aloof-twilight-67541 That could be problematic because of this issue which hasn't been resolved yet: https://gitlab.com/meltano/meltano/-/issues/2356 If you're using Meltano with Airflow, Airflow is already configured to not allow overlapping runs: https://gitlab.com/meltano/files-airflow/-/merge_requests/3
@aloof-twilight-67541 Would you be interested in contributing a fix for that first issue? 🙂 I'll help you figure out where the changes should go, of course.
a

aloof-twilight-67541

10/29/2020, 5:10 PM
I am not using Airflow, setting up cron jobs. I am not sure if I'm qualified enough to contribute a fix. More of an analyst than developer. But I can take a look!
r

ripe-musician-59933

10/29/2020, 5:18 PM
@aloof-twilight-67541 How's your Python? 🙂 I've left some notes on implementation in https://gitlab.com/meltano/meltano/-/issues/2356#note_438523204.
a

aloof-twilight-67541

10/29/2020, 5:21 PM
I can work in Python. I'll see what I can do
r

ripe-musician-59933

10/29/2020, 5:21 PM
@aloof-twilight-67541 Thanks! If you can create an MR, I'll gladly help you polish it up and get it over the finish line
a

aloof-twilight-67541

10/29/2020, 5:23 PM
will do!