Meltano

Meltano_Pipeline_Deployment.jpg

We have Kubernetes Deployment where we deploy multiple meltano pipelines, each with its own copy of `meltano.yaml` . All these pipelines connect to one common Airflow Postgres Database and Meltano Postgres Database. Both pipelines have their own schedules. We are facing issue where the schedule A is overwritten in Airflow whenever the pipeline B is deployed.  Both pipeline have common Airflow DB url environment variable

Does anybody have thoughts on the deployment strategy and why it could be overwriting the schedules ? What names/values would have to be unique so as to not have its values overwritten in PostgresDB ?

Hi <@U06CBKHRU9H>. If you have schedules with the same names in different projects, that would cause the dags to have the same ID and thus collide in the airflow db

oh great I will check and let you know <@U06BX9QQ4A3>

<@U06BX9QQ4A3> this is still not working, All the values are unique between 2 pipelines, but still the second pipeline does not get scheduled/orchestrated