What are the best practices with Meltano on EKS? Right now I have it deployed to ECR and an Airflow DAG pulls the image into the cluster. Can I scale to multiple pods if my workloads are quite large?

We’ve been running Meltano in EKS for about 18 months now, using Argo Workflows. It’s worked great for us. There were a few use cases early on (notably google analytics: https://github.com/meltano/meltano/issues/7165) where we used the Argo Workflows “fan-out” pattern to scale to multiple pods, but generally we’ve had no issues with a 1:1

meltano run

command : Pod ratio.

Awesome, thank you!

Would fanning out possibly improve speed?