Hi, I'm orchestasting Meltano with Airflow. I configured a extractor for X tables. For transparency and maintain communication per each table. I'm running in parallel, as a group in Airflow, iterating over the X tables with

--select

I use the same method for FULL and INCREMENTAL tables. I'm checking that sometimes the tables configurated as FULL incurs in

Another '2021-10-25T094706--tap-mysql' pipeline is already running

The error has sense because if Airflow trigger the command to initialize at the same time minute, Meltano job_id would be the same. job_id only registries until minutes. Any suggestion to avoid this? 1. I'm thinking add job_id for FULL tables, for difference each other. 2. Change the downstream in Airflow, from parallel to sequential.

Hi @boggdan_barrientos. I'd add the

job_id

to the full table tasks. Seems like a good solution without downsides