How do I create Airflow dependencies between Meltano schedules? e.g. schedule2 doesn’t run concurrently with schedule1 (they share resources)

I've only had to worry about concurrency with dbt runs. Other than that I haven't had much of an issue running things together. I'd assume airflow you can do something to that nature, maybe tieing a dag together so 1 -> 2 always is the order or something. (I'm guessing here)

the issue is that they’re using the same replication slot and I’m too lazy to go make more lol

https://stackoverflow.com/questions/51600181/airflow-locking-between-tasks-so-that-only-one-parallel-task-runs-at-a-time maybe?

@visch that could be an approach too but whether its limiting parallelism, creating dependencies, or any other feature of Airflow, I guess my question is really “how do I do that in

meltano.yml

?” I’m not running a standalone Airflow, I’m using

meltano invoke airflow

You have to add a lock somewhere, could be in your database, could be at the orchestrator (meltano just runs an airflow instance from the orchestrate folder I believe so you should be able to do the stack overflow idea)

I don’t see where I would do that, the orchestrate folder does not contain my dags

I don't understand what you're saying anymore sorry. Maybe someone else can help.

I guess I’d like to pass more Airflow parameters through Meltano’s schedule CLI rather than separately go through the Airflow UI itself

😄

@douwe_maan are there config files, either

meltano.yml

or others, where I can customize things like dag dependencies, assign dags to pools, etc. when using

meltano invoke airflow

?

@dave_lyons No, right now all of that needs to go into the DAG generators in the

orchestrate/dags

directory, either the default

meltano.py

one you can edit, or a new one