Question about running meltano in general, if I would like to create Meltano ETL with some config separation in mind. e.g. I would like to to have all the extracts and loaders but the configuration would be different based on some rule. can you accomplish this by creating different projects, environments or something. what has folks done?

I accomplished this in different environments

Hi @albert_m we manage this scenario with environment variables; we were generating a state id to pass to 'meltano elt'. In our setup, the config is all set outside meltano.yml tap and target settings. There are multiple opportunities to configure this setup now - including meltano environments. We've found Meltano environments have become necessary for 'meltano run' unique state ids - so perhaps that's the best place to start for a small set of different stages.

@aaron_phethean I need to learn more about meltano env. Any pages or links you Suggest to read about it?

Hi @albert_m I couldn't find any detailed documentation, this was pretty light https://docs.meltano.com/guide/production#managing-configuration. @aaronsteers had a good doc that explained the layers which would be great reference

@aaron_phethean thank you so much.

We have an unofficial and internal guiding doc here which we shared in an office hours I believe. Some of these statements and flows though are still a WIP based on what we think is the desired/optimal end state. If you can reference the official doc, that would be preferred but this should be interesting for discussion and I think (?) this the doc @aaron_phethean references above.

Thanks @aaronsteers that's the one 👊 Thought a mental model of the config layers might help @albert_m decide the best strategy for their project.