Any thoughts on resource management of pipelines running simultaneously. I am planning a system running pipelines of different customers, so would like to ensure fairness, no starvation etc. I am hoping someone has done this with Meltano - the learnings would be useful.

@subhash_gopalakrishnan I don't have Meltano-specific tips here, but Airflow's Kubernetes support may go a long way: https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html https://gitlab.com/meltano/files-airflow-kubernetes/-/merge_requests/1/diffs shows how you can use the

KubernetesPodOperator

with a Meltano project's Docker image (https://meltano.com/docs/containerization.html)

Thanks! This helps

I'd recommend using the kubernetesexecutor over what I've described above now. That way you can use the existing airflow dag generator. A combination of K8's ResourceQuota and PriorityClass resources should help you achieve what you're after too.

@niall_woodward do you know how to setup pods to be able to be rescheduled if a resource is overloaded? I set up autoscaling in GKE, but it doesn't actually autoscale because I'm not sure how to let the autoscaler/GKE know that the pods running Meltano can be rescheduled

See https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node

Thank you @niall_woodward! I was looking all over for this

@monika_rajput this might help for Airflow and Kubernetes orchestration @kai_yokoyama @tom_mcgrail @gunnar