Just finished watching today’s Office hours Zoom on Youtube. I wanted to add some comments about this, We’re couple of months in with Meltano development. We love it so far @amanda.folson But still I am kind of confused with Meltano + Airbyte relation, Upon reading the docs it appeared as if schedules of meltano pipelines HAD to be done and written in

meltano.yaml

file. I derived there is no other way except to maintain meltano’s internal airflow to schedule pipelines I feel the documentation could really shine with some additional “Deployment Best Practices” or “Common Deployment Patterns” Section 1. Deploy ELT project with inbuilt database (SQLite3) 2. Deploy ELT project with external Database 3. Deploy ELT project on Docker/Docker Compose 4. Deploy ELT project on Kubernetes with internal workflow orchestrator 5. Deploy ELT project on Kubernetes with external workflow orchestrator (Other than Airflow) It probably also needs a disclaimer that the

orchestrator/meltano.py

file serves as a starting point for building Airflow dags but is not really necessary to package it. When democratized data teams have more eyes on data flows, it is imperative to have a dedicated workflow orchestration cluster IMHO, Dagster.io and Prefect.io appears to give a better developer focused lifecycle for building and orchestrating workflows. But I have some confusions regarding how I can peel away from our embedded Airflow and use rather Dagster or Prefect. I’ll contribute with my experiences as I move forward, but hoping that this gets more eyes.

@binoy_shah Thanks for the feedback, much appreciated! https://docs.meltano.com/guide/orchestration#using-an-existing-airflow-installation touches on some of this, but definitely doesn’t make it clear enough that using

orchestrator/meltano.py

is completely optional, regardless of whether you’re using Meltano-managed Airflow or an existing instance.

Thanks for the feedback! We actually have a Docker tutorial coming out in the near future that I'll be sure to send your way if you want it. I agree that we need more deployment examples and we have those on the roadmap. I'll add the ones you listed here to the list if they aren't there already

Thanks for the links and updates Douwe and Amanda. It will surely be helpful for future deployments DThanks for the links