Hi everyone,
I'm new to the meltano game, and am ...
# infra-deployments
Stéphane Burwash
04/04/2022, 9:12 PMHi everyone,
I'm new to the meltano game, and am currently trying to create a CI/CD pipeline to automatically deploy our meltano project to our AWS environment. We currently do not have any framework to persist our states between build, and therefore have to re-sync everything at every build.
Would anyone have suggestions on best practices to achieve this?
Stéphane Burwash
04/04/2022, 9:13 PMI read this blogpost https://meltano.com/blog/deploying-meltano-for-meltano/ which gave me some insights, but I'm new to terraform so I just want to confirm this is the way to go
v
visch
04/04/2022, 10:11 PMThe best practice is a production grade DB. So RDS Postgres, point meltano there.
Another approach would be rds postgres serverless, another is to persist a local sql lite db between CI runs.
visch
04/04/2022, 10:11 PMTerraform is an option, I'm not sure I'd introduce it in an environment just for Meltano but 🤷 to each their own!
s
Stéphane Burwash
04/05/2022, 1:08 PM@visch Awesome thank you so much! We may go the terraform route since my boss is pretty savy with it, but it's good to know we have options!
And (this may sound dumb) but when we talk about keeping the current state, what do we want to keep? Like a copy of our .meltano folder?
v
visch
04/05/2022, 1:08 PMNot dumb at all!
https://docs.meltano.com/concepts/project#system-database
visch
04/05/2022, 1:09 PMThis system database stores the state for you, this explains this in more depth!
s
Stéphane Burwash
04/05/2022, 3:45 PMThank you so much! I'll keep you updated on how it goes 😄
Stéphane Burwash
04/05/2022, 7:38 PMUpdate: So currently I'm looking into setting up an EFS backend to store persisting data. We're currently running our meltano in ECS using docker.