are there any guidelines for scaling out Meltano? ...
# best-practicesr
rick
08/05/2021, 1:54 PMare there any guidelines for scaling out Meltano? production-based instructions just walkthrough a single server/container from what I can tell
k
ken_payne
08/05/2021, 3:42 PMHey Rick 👋 I gave a tour of our setup at tails.com at Demo Day few weeks ago that you might find interesting. We treat meltano pretty much like any other app/service; branch based deployment through CI/CD onto kubernetes.
In prod we run one kubernetes pod each for the airflow UI, airflow scheduler and the meltano UI processes. Database is AWS RDS. We also use the KubernetesPodOperator in Airflow in stead of the default executor so that each of our invocations of
meltano elt ...ken_payne
08/05/2021, 3:45 PMHoping to get some time towards the end of the month to get a generalised version of what we have out for the community to try 🙂