For anyone working with Meltano wrapped in Dagster, just curious what your deployments look like in terms of single or multiple instances. For example, before integrating Meltano with Dagster, I had three separate Docker Ubuntu instances (dev, staging, production), each pulling their respective built Docker images from DOCR on deployment via GitHub Actions. Hindsight being 20/20, this seems like overkill, especially with Dagster as I can configure branch deployments and knowing that I can run jobs concurrently. Note: Using a Hybrid Dagster+ deployment as opposed to the OSS.

Hey Steven 👋 Using branch deployments makes sense instead of having multiple instances. Depending on how much your environments differ, configuring separate environment variables for prod and branch deployments might get unwieldy.

I have dev and prod instances on Azure, using a single container deployment on Azure App Service. I have tried Container Apps in the past, but moved to App service as this gave better performance for less costs. ACA caps out at something like 4g by default, and now I have 16gig available. In terms of pipelines, if I am making changes to meltano, I have a docker build and push step (which installs all plugins), and then a manual step to reload App service, as I don't want to interrupt a long running job. I also have dbt models on top of meltano streams, and these I deploy separately to a storage volume, so I can reload the repository in dagster without taking down the whole container.

@Andy Carter Thanks! So you're running DBT outside of Meltano and directly in Dagster?

yes that's right. I install dagster as a utility in meltano, and include dbt in the env definition. Then I start my container with

meltano invoke dagster:dev

which is a custom command.

We run Dagster OSS in Kubernetes. We have a meltano project set up as a Dagster code location using the

dagster-meltano

library, and we have a dbt-core project set up as another code location using

dagster-dbt

. We have six application environments run Dagster in six Kubernetes clusters. We’re using the dagster/dagster and dagster/dagster-user-deployments Helm charts - we inflate them in ArgoCD and use lots of kustomize templating to scaffold out all the overlays we need.