I noticed that there is a Meltano Helm chart which is incomplete and that an older blog post mentions using the airflow helm chart as a base, but has anyone used that helm chart to deploy Meltano? We're (@Stéphane Burwash) looking to migrate completely to kubernetes so we can scale the runs.

cc @ken_payne I think you were the one that wrote the original helm chart if I'm not mistaken? We're trying to run an airflow scheduler and the webserver through meltano within Kubernetes

I'll let Ken chime in on that but are you all using ECS Fargate right now? I saw https://github.com/potloc/meltano-ecs-fargate-terraform-example. What airflow executor/operators are you using today? Have you considered staying on Fargate but implementing the same pattern that Ken used for our kubernetes deployment in that blog post? i.e. a single worker manages spinning off a new container per task

Basically, we are currently migrating from AWS FARGATE to GCP for hosting meltano. Currently our executor is the local executor and we are trying to graduate to the kubernetes executor while migrating to kubernetes. That's a good suggestion! @nicholas_lockhart what do you think of the blog post?

Goal is for a single instance scheduler to spawn and manage multiple containers and run their DAGs and the webui as the interaction for @Stéphane Burwash We're looking at kube as a method of infinte scaling through GKE autoscaling. I have the most experience in that. I read through that blog post and it is mostly what I am looking at from an infra perspective.

That makes total sense, I'm less familiar with GCP. In the past I built the same infinite scaling type workflow with Meltano/Airflow/Fargate so wanted to make sure you werent switching to kubernetes only for that. Our kubernetes deployment based on the blog post and helm chart has been working really nicely though. This is our terraform for deploying https://github.com/meltano/squared/tree/main/deploy and airflow pieces https://github.com/meltano/squared/tree/main/data/orchestrate/airflow if its helpful.

The blog post is helpful, but I am finding that the Chart isn't connecting the scheduler to the webserver. It seems obfuscated, which is why I was curious about using the official Airflow chart instead and whether meltano could be strapped into the container and run on that chart