For those using airflow to schedule meltano (separ...
# best-practicesd
devon_seitz
02/10/2023, 3:01 PMFor those using airflow to schedule meltano (separate airflow from meltano maintained), do you have meltano in a separate repo / docker from airflow?
devon_seitz
02/10/2023, 3:03 PMwondering how to handle dependency conflicts between airflow, meltano, and potentially dbt
s
Stéphane Burwash
02/10/2023, 3:41 PMWe have everything in the same docker project 😄 we run dbt & airflow through meltano, so we don't really have dependency issues
d
devon_seitz
02/10/2023, 3:47 PMAhh yeah we cannot unfortunately so hoping to figure out what best practice is when they are separate
My main question would be, if they’re in separate containers, how do you trigger the sync - probably through the API?
s
Stéphane Burwash
02/10/2023, 3:47 PMTrigger a sync as in trigger a run?
Stéphane Burwash
02/10/2023, 3:47 PM(Of airflow)
d
devon_seitz
02/10/2023, 3:49 PMTrigger a meltano sync
devon_seitz
02/10/2023, 3:50 PMSince they’d be in unrelated environments - i’d imagine we’d have to call the meltano API since we couldn’t just use the CLI
devon_seitz
02/10/2023, 3:50 PMor perhaps a dockeroperator
s
Stéphane Burwash
02/10/2023, 3:55 PMYeah that could probably work 😄 I have never had this issue, so sadly I can't speak to it. I'm fairly confident though that it is common place in the community to have a separate airflow integration from your meltano one, so someone should be able to help you out! Sorry for not being more helpful
l
luis_ventura
02/10/2023, 7:02 PMI have deployed meltano along with Airflow in the same docker compose project but as different docker compose services that are in the same docker network. The Airflow worker communicates with meltano using ssh.
I tuned the
meltano.pySSHOperatorBashOperator