Hello everyone I have a question Is it possible to...
# troubleshooting
j
Hello everyone I have a question Is it possible to trigger a Meltano ETL from a remote (meaning : another server) Airflow ?
v
Out of the box with airflow I think the answer is no. But you can setup airflow to ssh to a box and call commands. Maybe using ansible or just direct ssh. From a quick look at the Executor section Celery, and Dask could help too but if you're not suing them a "simple" ssh script might do the trick for you for now
j
Thank you very much @visch 🙏
v
Note I have never been an avid Airflow user but I've used a bunch of scheduling tools 🙂 and looked into Airflows docs more than once deeply
j
No but the this is that Meltano considers that scheduling happens directly in the same process which it rarely does
v
I don't follow you I guess, I'm missing something probably?
j
I mean when you install Meltano, it comes with an Airflow integration for orchestration. But what if (and that’s the most likely) you already have an Airflow running ?
v
https://meltano.com/docs/orchestration.html#using-an-existing-airflow-installation , it looks ike you can copy the dags from the orextrate/dags folder and put them on your existing Airflow server. By running
Copy code
meltano add files airflow
If you wanted that would work to it looks like!
j
I’ll take a look at it
Thank you very much for your time and insights @visch
v
No problem let me know if that works, I'm curious!