:melty-bouncy: Howdy Meltano community! :melty-bou...
# best-practicesu
user
10/11/2022, 1:23 PMmelty bouncy Howdy Meltano community! melty bouncy We care about your knowledge & opinions. So: How do you deploy your meltastic Meltano? Add your own setup, we certainly don't know all of them!
s
Sven Balnojan
10/11/2022, 2:20 PMI'm learning something new every day. @pat_nadolny we are running two different kinds of deployments? Or do you have your own little Meltano fun setup?
Sven Balnojan
10/11/2022, 2:28 PMBtw. a follow up question to anyone who feels like it and @thomas_briggs in particular: How do you get a new version of your meltano project repository onto the instance? Do you install git on it? Do you use some CI action and do some file replace magic?
p
pat_nadolny
10/11/2022, 2:34 PM@Sven Balnojan before joining the Meltano team I used Meltano at an ecommerce company and we deployed it on ECS. We had a standalone (not managed via Meltano) Airflow service running there too which kicked off the meltano jobs as ECS tasks using the EcsRunTaskOperator.
t
thomas_briggs
10/11/2022, 2:35 PMYeah we use git to pull down the main meltano.yml (and some utility scripts too, e.g. for finding the latest logs)
d
dylan_just
10/11/2022, 11:45 PMAWS Batch is awesome for running meltano. We're running meltano elt in it daily.
s
Sven Balnojan
10/12/2022, 5:57 AM@dylan_just that sounds good, AWS Batch is even simpler than using ECS, right?
d
dylan_just
10/12/2022, 6:01 AMA little bit, yeah. Since it's just a periodic task, you don't have to worry about keeping an instance alive. Also, you can use Fargate for compute without worrying about cost. With ECS, you normally want to make your own EC2 clusters for cost.
dylan_just
10/12/2022, 6:02 AMHow do you get a new version of your meltano project repository onto the instance?Immutable deployments, baby. CI builds a docker image with the meltano project, CD rolls it out.
s
Sven Balnojan
10/12/2022, 10:19 AMQuestion to @david_wallace @lidong_zhao @boggdan_barrientos: I'm curious to know how you handle state storage?
j
jonas_kalderstam
10/21/2022, 1:41 PMcertainly don't think of Kubernetes as "heavy". rather one of the lightest approaches.
We run Meltano in Kubernetes Cron Jobs
l
lidong_zhao
11/03/2022, 3:03 AM@Sven Balnojan Sorry, just saw your message, we use PostgreSQL for the Meltano database
s
Sven Balnojan
11/03/2022, 7:21 AM@lidong_zhao Thanks for the feedback! FWIW, I've also heard of one alternative for handling state: to pass the state locally as some kind of injected artifact.