Hi I am checking the structure of different ` meltano yml` i Meltano #getting-started

Hi, I am checking the structure of different `*.me...

mindaugas_nizauskas

05/05/2023, 5:58 AM

Hi, I am checking the structure of different

*.meltano.yml

in https://github.com/meltano/squared/tree/main/data repository. 1. Are you splitting into multiple meltano

.yml

files manually? 2. When adding a new extractor or loader, are you using using

--interactive

mode only?

--interactive

mode and later manually cleaning

.yml

files? Manually updating

.yml

only?

pat_nadolny

05/05/2023, 12:36 PM

@mindaugas_nizauskas I'm the primary contributor to that repo. Yes if I add a new plugin it gets placed into the top level meltano.yml and I use

--interactive

but once I've got it working I re-organize manually. I think theres an issue somewhere for meltano being smarter about where to put configs when the yamls are split. Also to be clear theres no "recommended" way of organizing your yaml files if you split them, the way its organized in the Squared project is just the way that I liked to organize them. Honestly I'm not in love with it though, especially how my environments are in separate files

mindaugas_nizauskas

05/05/2023, 12:44 PM

How would you do it, if you start from scratch? 🙂

pat_nadolny

05/05/2023, 1:48 PM

Heres a thread that might be helpful https://meltano.slack.com/archives/C01TCRBBJD7/p1666266979470469

pat_nadolny

05/05/2023, 1:54 PM

I'm not totally sure how I'd do it if I started over and its pretty easy to change along the way so youre not locked into making the decision now. I like the idea of separate files per plugin and like the centralized aspect of how I have a standalone file for each environment. I could imagine that putting the environment configs in each plugin file would get hard to manage, if I need to make a change across many files. Maybe something like an

environment_override.yml

for each plugin type so I can centralize similar configs like: •

extractors/tap_slack.meltano.yml

•

extractors/tap_github.meltano.yml

•

extractors/environment_override.meltano.yml

•

loaders/target_snowflake.meltano.yml

•

loaders/target_jsonl.meltano.yml

•

loaders/environment_override.meltano.yml

I'm just brainstorming here, I havent fully thought it through

Open in Slack

Previous Next