https://linen.dev logo
Join Slack
Powered by
:wave: hey everyone. We manage multiple different ...
# announcements
j
👋 hey everyone. We manage multiple different projects >20, that each have their own datasources. Is there an easy way to segment different projects (including scheduling)? Or should I take the approach of instantiating 20+ different Meltano instances in different directories (or deploy 20+ instances onto Kubernetes). How do we orchestrate Meltano itself?
d
@john_g Hi! Are you interested in just running the pipelines, or also having the UI available for all of these projects?
j
Running the pipelines is most important, but UI would be amazing if reasonably possible.
d
@john_g All right, I'm sure we can figure something out 🙂 I'm about to break for lunch, but I'll get back to you after!
j
Thanks a lot Douwe, I really appreciate it!
d
I think you're going to be best off setting up a single Airflow instance, and modifying the default `meltano elt` DAG generator to not just create DAGs for one project's 
schedules
defined in 
meltano.yml
, but to iterate over all projects in a specific directory (or hardcoded list), and repeat the process for each. Each project could be its own Git repo, and you could have one master project with that main Airflow DAG, pulling in all other projects as submodules. Then adding a Meltano project (which I imagine would happen sometimes?) would be a matter of spinning up a new Meltano project, pushing it to Git, adding it as a submodule to your main project, and then deploying it so that the production Airflow instance will rerun the DAG generator and pick up the new projects
If you're going to want to use the Meltano UI, you are still going to have to run 
meltano ui
in each project and manage those instances separately, but for the moment you may not need the UI at all and the Airflow UI may be sufficient
@john_g Does that sound feasible? I don't know too much about your specific use case and requirements, but I'd like to help you figure this out! Happy to jump on a call too if that would be useful.
Once we have https://gitlab.com/meltano/meltano/-/issues/2048, you could also use those images with Airflow's 
DockerOperator
rather than the 
BashOperator
that's used today. Each of your projects could have its own CI pipeline to generate the new docker image, that your Airflow DAG could reference by image URL/path/name. It'd be easy to run 
meltano elt
that way, but if you have your schedules defined in each project's 
meltano.yml
, there's not currently a way for your master Airflow DAG generator to pull those out of each image using the CLI, so you wouldn't be able to use 
meltano schedule
and instead would need to define the DAGs for each pipeline "manually" in your master project. It wouldn't be too hard to add a new command to list all of a project's schedules in a way that your master DAG generator could parse, of course, if you decide to go in that direction. For the moment, I'd stick with the 
BashOperator
and 
meltano elt
though, like we do in the default DAG generator.
j
Thanks a lot Douwe. I'll start along this path and stick with one Airflow as well.
d
@john_g Sounds great! If you'd like to chat about it some more, feel free to book a slot on my calendar through https://calendly.com/douwem/half-hour-meeting 🙂
j
👍
Open in Slack
PreviousNext
Powered by