Hey team (wow 3 posts in a row 😐 ) - looking for some general best practices feedback re: deploying meltano within a production environment for use with multiple pipelines which can be self serviced by other teams etc. 01. Centralised scenario

- Primary meltano instance runs all pipelines and jobs for all the things
- Teams adopt taps/targets their specific configurations 
- Those taps/targets are added to the main instance and specific pipelines with (inherited) configs for that team so taps 
- centralised logging, alerting etc
- always up - configurations mounted to some location (eg a deployment connected to a github repo for the yml )

pros: 
- observability centralised
- makes the ui useful
- lower overhead for maintenance

cons: 
- build failures / job failures may impact other jobs (? I havent read up on this)
- higher associated risks for failure of system 
- meatier instances required

^ is this scenario best practice? When working with our current k8s infra, I’m feeling the pull of deploying single pipelines and across multiple pods, since I can just fire them up via cron (below) 02. Distributed scenario

- Multiple meltano instances created for each pipeline (eg Bigquery -> S3 pipeline)
- each instance controls a single requirement for data movement. 
- state still associated with some external db as above. Possibly in separate tables
- option to basically run on demand - using cron to trigger the jobs

pros:
- lighter, simpler to deploy, can trigger via cli commands with basic cron jobs
- risk distributed across multiple instances. If stuff breaks on 1 instance, others are unaffected


cons:
- Observability will need to be across x instances deployed (additional work to get this)
- UI basically useless since there will likely be only 1 pipeline/job in each

Best practice feels like centralised from a long term perspective, but from a POC/MVP perspective i’m feeling like it’d be much easier to deploy with scenario 02. Thoughts and feedback much appreciated here.

I'd lean towards central orchestration (Airflow / Dagster / K8s cron) , distrubted meltano runs per elt job. Gives you centralization for scheduling / logging, and decentralization for the individual jobs

I had a very similar deployment to scenario 2. A single Meltano image was built during CI/CD, then multiple k8s cron jobs with different

meltano elt ...

commands. It is nice until it isn't and a job fails and you miss having the history and UI of something like Airflow. So, like @visch said, if I had the resources to leverage something like Airflow + K8s operators, I'd go for that.

Thanks guys! I’m quite surprised the distributed scenario is more adopted. Really grateful for this feedback. I’ll bring this up with the team this morning. With this approach in mind, it feels like once you start scaling and orchestrating, the meltano ui becomes much less of a draw. This is interesting, since I was visualising the ui as a centralised alternative to an orchestrator if companies didn’t adopt that.

I wanted to jump in here and highlight that I’m thinking (and probably need to formalize in an issue) of how we can best support a distributed scenario where there are multiple instances of Meltano running. I want there to be a useful UI where you can see any/all instances of Meltano running. We should be able to support a happy path where you can get up and running and have all pipelines running with one project, but we should be able to support the distributed use case as well

@taylor I agree that a UI with all instances running, (distributed or not) would be fantastic. The gotcha i’m personally hitting at the moment (might be due to lack of understanding Meltano enough) is creating new pods for each run (then shutting them down).