Hey @douwe_maan, I looking more into how I can use meltano without refactoring my current setup from scratch. Is it possible to use meltano for EL in a larger dag that I already have. The DAG runs on airflow, so an easy way to integrate would be some kind of meltano airflow plugin. I know one of the important goals of Meltano is making singer taps easy to work with and provide tooling for running them in production. I am down to swap some of my own bashoperator magic to use Meltano airflow operator today but refactoring current production and host another airflow meltano project seems like a bigger task. Let me know what you think

Is it possible to use meltano for EL in a larger dag that I already have. The DAG runs on airflow, so an easy way to integrate would be some kind of meltano airflow plugin.

For sure! You should be able to use the

meltano elt

command with the

BashOperator

. In fact, that's exactly what the default Meltano DAG generator does, that you get if you run

meltano add orchestrator airflow

: https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py

My gut feeling is making a airflow native operator will allow data engineers to leverage al lthe other features of Airflow like xcom /extending it for their use cases but bashoperator is a simple enough way to get started for sure. That makes meltano a blackbox almost. That is a nice abstraction