Hi all, I am just trying out Meltano (after tried some other similar tools). So far I am quite impressed and want to assess what it would take to put it in production in the company I work for Currently we have a running Airflow instance on Kubernetes, and use KubernetesPodOperator for a lot of tasks. When looking at Meltano, I am struggling a bit to understand how it fits in to this structure and what would be considered best practice? I can see these possibilities, but do you have any thoughts? 1. I can containerize my connections and run it with KubernetesPodOperator in Airflow (meltano elt …) - meaning I would not use Meltano scheduler and UI would not be available 2. I can use Airflow as orchestration with “always on” instance Meltano running separately. Use dag-generator (not exactly sure how this add dags to my airflow instance or how to install? I had a look at airflow-files but not much wiser since) 3. Something else? I hope my questions were somewhat clear. and thanks for the great work!

Hey @emil_nilsson! Within Meltano (the company) we have our own Meltano instance that uses Airflow on Kubernetes also - @ken_payne wrote up a good blog post about it how we chose to do deployments, that might help. And the repo is https://gitlab.com/meltano/squared

Thank you @pat_nadolny! I will go through the article and repo. We have a on-premise k8s cluster and don’t use terraform as such. But as far as I understand it, meltano executes dag-generator given schedules and puts it into airflow dag-folder (either by pointing airflow to meltano or the opposite). Is that correctly understood?

@emil_nilsson the dag generator can be thought of as just another dag that Airflow consumes. You just place that generator dag file in the dags folder that Airflow is reading from. In the case of Airflow installed using Meltano, Airflow is already configured to look in the

orchestrate/dags/

directory but if you have your own Airflow instance already running you just drop that dag generator into Airflows dag directory. Usually people write Airflow dags by hand but with meltano the dag generator asks meltano (via CLI subprocess commands) what schedules are set in your meltano project, then it builds tasks and registers a dag for each schedule. Note - the generator needs to have access to the meltano project and have meltano installed so it can run meltano cli commands to get the schedules, Ken describes how to solve this with k8s in the blog. Does that make sense?

That sure does make sense. Thank you for your descriptions and time spent

Old thread, but this may be useful for anyone finding this: It is not necessary to install

meltano

on the scheduler, in fact the squared repo just parses the yaml-file directly https://github.com/meltano/squared/blob/9cf437c4ec811c850f37a1f9bb69911487e9e836/data/orchestrate/dags/generator.py#L42