Just finished watching today’s Office hours Zoom on Youtube. I wanted to add some comments about this, We’re couple of months in with Meltano development. We love it so far @amanda.folson But still I am kind of confused with Meltano + Airbyte relation, Upon reading the docs it appeared as if schedules of meltano pipelines HAD to be done and written in

meltano.yaml

file. I derived there is no other way except to maintain meltano’s internal airflow to schedule pipelines I feel the documentation could really shine with some additional “Deployment Best Practices” or “Common Deployment Patterns” Section 1. Deploy ELT project with inbuilt database (SQLite3) 2. Deploy ELT project with external Database 3. Deploy ELT project on Docker/Docker Compose 4. Deploy ELT project on Kubernetes with internal workflow orchestrator 5. Deploy ELT project on Kubernetes with external workflow orchestrator (Other than Airflow) It probably also needs a disclaimer that the

orchestrator/meltano.py

file serves as a starting point for building Airflow dags but is not really necessary to package it. When democratized data teams have more eyes on data flows, it is imperative to have a dedicated workflow orchestration cluster IMHO, Dagster.io and Prefect.io appears to give a better developer focused lifecycle for building and orchestrating workflows. But I have some confusions regarding how I can peel away from our embedded Airflow and use rather Dagster or Prefect. I’ll contribute with my experiences as I move forward, but hoping that this gets more eyes.