I'd like to report we are using meltano for extrac...
# announcements
a
I'd like to report we are using meltano for extracting and loading data in a production environment. The configuration is working well to manage selected streams and fields. State is stored and applied automatically. Taps and targets are streaming with each other. Thanks @douwe_maan and team for your hard work. Hoping to see much more!
a
Where did you deploy it out of interest?
I'd be interested to know how you handled setting up Meltano's own db, as I'm just looking at changing that from sqlite to Postgres now
d
That's really great to hear @adam_roderick! Looking forward to more feedback, feature suggestions, and contributions 😉
a
@al_whatmough We use prefect as our orchestration layer, and everything runs on AWS. Meltano can connect to any postgres database, so however you set that up is up to you. We use terraform to automate our infrastructure deployment, which now includes a small, dedicated postgres DB for meltano.
We actually use the same postgres DB for multiple environments (staging, prod, etc), because all our job_ids have an environment tag, which keeps runs separate from each other
d
@adam_roderick Clever! As long as you're not storing configuration in your system DB, you shouldn't run into any issues with that approach
a
Ah thanks for pointing that out. We are only storing config in meltano.yml and in env vars
We found that to be easier to deploy
d
For sure, I wouldn't recommend using the DB for settings storage unless you're using the UI as your primary way of interacting with Meltano and configuring plugins, which is definitely not a recommended workflow today, but may be in the future