im searching for how to connect airflow which is i...
# best-practices
e
im searching for how to connect airflow which is installed in a container and meltano pipelines .. I can successfully use the internally provided airflow, but now needing to connect downstream data transformations to the raw data coming out.. and that feels like it should not live in the tap ..
could it be just as simple as.. running an airflow DAG that.. triggers
meltano run
.. waits for the results.. then takes say the parquet data and runs other jobs written in JuliaLang , python , etc... ?
https://docs.meltano.com/guide/orchestration searching this doc , so far..
perhaps the option is to .. locate and run the DAG created by Meltano in the meltano project
maybe I ditch the stand alone airflow idea and poke around this https://docs.meltano.com/guide/orchestration#using-airflow-directly use the meltano managed airflow to run my transformation jobs .... somehow