Hi everyone, we are considering running Meltano either on AWS ECS or on an EC2 instance. Does anyone have any ideas about this and is this documentation page still up-to-date and what additional steps would you recommend? https://meltano.com/tutorials/install-on-aws.html#final-steps Thank you in advance for your support!

Personally I am using meltano "from scratch" without Docker.. just to get started.. but in general I am going to deploy it with Juju Charm. This way I can bake a config and test locally.. then deploy that config sort of like cloudfoundation and terraform allow .. if you go with the docker container approach I imagine it'd be just as similar deploying this.. as deploying any flask or python based webapp. I still need to test apache-airflow.. but that is likely also going to be containerized by myself.. but you might have different plans?

@ole_bause I recently deployed Meltano using ECS Fargate + Cloudwatch scheduled task. Using github action for ci/cd and terraform for IaaC

Hi, I'm very interested in this topic as well. @ken_payne , I have seen your excellent presentations on the Kubernetes deployments. Do you have some similar presentations of more detailed guidelines for running Meltano on ECS Fargate with say an independent Airflow instance perhaps in a container or MWAA for the scheduling aspect. It would be nice to run the Meltano Container for a particular EL operation and then shut it down once the task is complete. I say this as not all organizations have the skills or support for a Kubernetes deployment. My organization would prefer at this stage to stick with a ECS Fargate deployment? Any insight would be much appreciated. Thanks Steve

Hey 👋 Also very excited for this discussion 🙌 I completely agree that k8s isn't ideal for everyone, and we are by no means looking to standardise on it 😅 I just happened to use it in my last two roles, so it is 'low hanging fruit' while I onboard. We created a new epic last week to detail a broader approach to supporting deployments on all the various infrastructure platforms, and it would be great to capture this amazing feedback there too. I had thought to move on to EKS on AWS (to make use of the minimal helm charts already developed for local k8s), but we could just as easily move on to ECS in stead if it's going to be of more use to the community. @taylor leads our feature prioritisation efforts and will be interested in this too 🙂 @keat we are hoping to use File Bundles (first example is local k8s) to allow users to add Infrastructure-as-Code (terraform, cloudformation etc.) templates to their projects. These will of course be open source and we would love ❤️ to have contributions from the community. If you have already developed templates and are interested in sharing them we'd be happy to help!

@ken_payne Unfortunately I don’t have one for k8s. But I have been procrastinating on my terraform-fargate-cloudwatch-module + github action deployment automation that I don’t mind sharing with the community 🙂

any recommendations on deploying a meltano schduler onto an existing ec2 instance? just a a simple dag in the

orchestrate

folder that runs a shell script daily. the idea is to deprecate the existing cron scheduler

@kevin you can review the install hook I wrote in bash https://gitlab.com/jrgemcp-public/charms/meltano-charm/-/blob/main/hooks/install if you want to skip juju just take that set of instructions on getting started sans docker

Hey @keat! Did you ever get chance to share what you'd been working on with deploying Meltano to ECS using Fargate? Would be super interested to take a look at it if you don't mind sharing! I'm looking at setting up a new Meltano project on AWS and would be keen to get some info on best practices. Cheers!