Hey, y'all! We've been trying out meltano and I have a few issues/MRs I plan to open but it would help to have some clarity on this first. Where do you recommend state is stored and what's the suggested workflow right now? Pipeline configuration seems to be stored both on the filesystem and in a database. Given that, should we: 1. use the server for storing filesystem and db state, and setup pipelines via commands on the server itself? Does this mean that we loose out on version control for dbt models, etc? 2. set up pipelines locally, commit the results, and deploy to the server? Does this mean we have to connect to the production database from our local environment? Do we run the server UI in read-only mode? 3. something else? The docker-compose file seems to suggest the first given the

./:/project

volume, although I thought that might be for targeting "no-code" user archetype

@julian_knight Welcome, and good questions! I'm planning to spend most of the day on documentation, especially around production usage, so your timing is also perfect in helping me figure out what isn't clear yet 🙂 The recommend workflow is to set up your entire project locally using

meltano

commands, and to manage it like any other software development product with Git etc. Non-sensitive configuration would go into

meltano.yml

(you'll need to add it manually until we get to https://gitlab.com/meltano/meltano/-/issues/2013), and (locally) sensitive configuration can go into the system database, which lives at

.meltano/meltano.db

by default, or

.env

, both of which are gitignored. When you deploy Meltano, you'll want to use a Postgres system DB instead of the default SQLite one (see https://gitlab.com/meltano/meltano/-/issues/2072), which is where pipeline state will be stored. Sensitive configuration (secrets) in production can be stored either in that same system DB using

meltano config

or

meltano ui

, or using environment variables, possibly in combination with a tool like chamber (see https://gitlab.com/meltano/meltano/-/issues/2038#note_346151006) to store the actual secrets in a vault

Thanks @douwe_maan! That makes sense. I don't have any other questions, but if you have any customer discovery-type questions for me I'd be happy to hop on a call

@julian_knight Would you have a few minutes right now? No super specific questions, but I'd love to know more about your use case and what you're thinking about Meltano so far!

@douwe_maan sure! could you give me about half an hour to wrap something else up?

@julian_knight For sure, no rush 🙂