Hey Everyone is there are a SaaS provider that hosts Meltano? I saw a post about it on the blog but it seems to have disappeared now. I'm just wondering what the best way to deploy this into production is (i've read through the docs and though it describes how to do it for both containerised and non containerised apps it's been a bit difficult to setup on my setup). I've setup our pipelines such that each MR creates an isolated k8s environment. The idea is that I can include meltano in the pipeline so I can run any tests etc on it. Or should I just forget that and instead deploy a single instance of meltano on GCP for our production environment and leave it? Just FYI I am trying to do this with Mongo > Postgresql and running the ELT directly doesn't work, have to specify a

replication-method

which means invoking the tap-mongodb and editing the file before running the elt.

Hey Everyone is there are a SaaS provider that hosts Meltano? I saw a post about it on the blog but it seems to have disappeared now.

Hi @nathan_lee! There isn't currently, but I know that https://www.astronomer.io/ is evaluating offering Meltano on top of their hosted Airflow offering. For now we're focusing on self-hosting, though.

I'm just wondering what the best way to deploy this into production is (i've read through the docs and though it describes how to do it for both containerised and non containerised apps it's been a bit difficult to setup on my setup).

What did you get stuck on exactly? Because of the many different possible deployment environments and setups (with built-in Airflow, on an existing Airflow, without Airflow and using K8s cron, or with a different orchestrator entirely), the docs are relatively generic with a lot of the specifics left as an exercise to readers who are presumed to be experts on their current deployment platform of choice. Of course, we want to improve our documentation around common setups, especially on Kubernetes, but we need feedback and help from people like you to get there. For a start, check out #C01E3MUUB3J and https://gitlab.com/meltano/meltano/-/issues/2149, and please weigh in if you have any additional thoughts!

I've setup our pipelines such that each MR creates an isolated k8s environment. The idea is that I can include meltano in the pipeline so I can run any tests etc on it. Or should I just forget that and instead deploy a single instance of meltano on GCP for our production environment and leave it?

You can have review environments, and a production environment, right?

Just FYI I am trying to do this with Mongo > Postgresql and running the ELT directly doesn't work, have to specify a 

replication-method

  which means invoking the tap-mongodb and editing the file before running the elt.

Did you find https://meltano.com/docs/plugins.html#metadata-extra? One of my tasks for this week is actually to document that more clearly: https://gitlab.com/meltano/meltano/-/issues/2408

Thanks Douwe for the detailed response! In terms of the deployment I think I got stuck because it was so flexible, which is great, I just need to figure out like you said which kind of setup to go with. I’ll try to come up with a generic setup using cronjobs — And yes I could, I wasn’t really thinking enough about having different environments haha. — The new docs were super helpful, though I am missing one thing which is the “tap.projections” because currently is being treating as a nested selection. I see that it’s an issue and if you’re not already looking into it I’ll try to have a go at it this weekend!

Yep, there's an issue out for that one: https://gitlab.com/meltano/meltano/-/issues/2388