I’m searching around for examples of meltano usage in production, and I’d love be pointed to any examples you all know of. For context, we’re planning on using airflow as the orchestrator for the backend. Are most deployments like the examples / tutorials, that is, run everything on a single box?

Hi Paul, welcome! I know @adam_roderick is running

meltano elt

in production using Prefect.io, @julian_knight is running Airflow and Meltano UI on convox.com, and @nevin_morgan is currently looking into deploying pipelines on Kubernetes using a Helm chart: https://gitlab.com/meltano/meltano/-/issues/2149 Because Meltano is orchestrator-agnostic, running it in production comes down to running an orchestrator (like Airflow) in production, and then instructing it to run the

meltano elt

CLI, directly in the local environment (e.g. using Airflow's

BashOperator

) or using a containerized project's Docker image (using

DockerOperator

). Whether you're going to run everything on a single box or not will be a matter of how your orchestrator is set up and deployed, which does not necessarily affect how Meltano itself should be configured for production, which looks pretty much the same no matter what orchestrator or setup you're using, since it's just a project directory and a CLI in the end: https://meltano.com/docs/production.html If you haven't already, I suggest looking into how to deploy Airflow in production first (on which plenty of non-Meltano-specific material can be found online), which will also have you decide whether to use a single-node or multi-node executor: https://airflow.apache.org/docs/stable/best-practices.html#multi-node-cluster Then, deploying Meltano will be a matter of deciding whether you're going to containerize it (https://meltano.com/#containerization), adn following the best practices in https://meltano.com/docs/production.html.

Gotcha! Thanks @douwe_maan this is really helpful! I’ll check out the Airflow in production examples. (and double thanks for the fast response)

Good luck and let me know if I can help with anything else!