I was interested to see folks discussing Dagster as a potential alternative to Airflow. Has anyone looked into Flyte? It seems to be getting some big-name adoption (orignally @ Lyft, and Spotify just posted that they were moving from Luigi to Flyte). https://flyte.org/ I think you can do the equivalent thing as https://engineering.widen.com/blog/Dagster-+-Meltano/, namely run a shell command to do trigger the Meltano EL(T). However I think these DAG-oriented tools actually have a bit of an impedance mismatch with Meltano, since Meltano is not oriented around per-table tasks; you could make a case for Meltano generating a DAG of smaller work chunks; for example,

(tap-mysql table1 >> target-snowflake table1 >> dbt table1), (tap-mysql table2 > ...), ...

so that we don’t need to do a full

meltano elt

re-run if one of the extract/load jobs fails for

table1

. Wondering if there’s any thinking on a tighter integration from Meltano the DAG-oriented workflow that is inevitably required for larger-scale data pipelines?

We’re actually thinking about things like that! https://gitlab.com/meltano/meltano/-/issues/2920 And

meltano run

would be a good place for us to start putting stuff like that in 🙂 https://docs.meltano.com/reference/command-line-interface#run