https://linen.dev logo
Join Slack
Powered by
I don’t see an #orchestration channel, so I’ll put...
# announcements
s
I don’t see an #orchestration channel, so I’ll put this here. Is the Airflow that comes w/ Meltano limited at all? Or can I use Airflow’s entire functionality by calling 
meltano airflow [...]
? For example, the auto-generated 
DAG
from creating a 
pipeline
is awesome, but I’ll need a 
DAG
that has > 2 steps. Does this violate meltano’s current idea of a ‘’pipeline’? or is it compatible?
d
Is the Airflow that comes w/ Meltano limited at all? 
@skw Nope!
Or can I use Airflow’s entire functionality by calling 
meltano airflow [...]
 ?
Yes, as long as you don't forget the 
invoke
in 
meltano invoke airflow ...
😉 As it says under https://meltano.com/docs/orchestration.html#using-an-existing-airflow-installation:
In fact, all 
meltano invoke airflow ...
does is populate 
MELTANO_PROJECT_ROOT
, set Airflow's 
core.dags_folder
setting to 
$MELTANO_PROJECT_ROOT/orchestrate/dags
(where the DAG generator lives by default), and invoke the 
airflow
executable with the provided arguments.
You're not required to use the default DAG generator at all, it's just there for convenience if you want to use 
meltano schedule
🙂 Using 
meltano elt
directly using your own pipeline scheduling setup is perfectly valid as well.
s
Thanks @douwe_maan. I’m going to get a multi-task DAG setup today and we’ll see how it goes 👍
d
@skw Awesome, let us know if we can help!
r
@skw Just a note for sharing that we are thinking of using Meltano and Airflow little differently - Running a separate instance of Airflow which has DAGs to call different Meltano Containers in data flows using DockerOperator. My thinking is that generally you will have many ELT flows managed by separate Meltano containers and so it is better to configure a central Orchestrator to manage all ELT flows.
s
interesting. thanks. I will explore that as an option.
If you are using DBT - does your DBT instance operate across Meltano containers? Or does each have its own DBT pointing to the same materializations? @rahul_anand
or separate materializations entirely? Curious how you are structuring the end-target. Our project is a real estate analytics platform. We currently only provide data for a single county, but I am trying to formalize the process for adding additional counties. Each county has its own siloed listings, property data, etc., all in varying formats. I think a unique meltano instance for each county makes sense. Once data is extracted and loaded, we will use DBT to transform County A’s data to a table that also receives transformed data from County B.
6 Views
Open in Slack
PreviousNext
Powered by