https://linen.dev logo
Join Slack
Powered by
re: select lists for taps - I have a bunch of incl...
# troubleshooting
t
re: select lists for taps - I have a bunch of inclusions I want included in all environments (e.g. A*., B.*, etc.), and then a bunch of exclusions that are environment specific. I thought if I put the inclusions in the plugin definition and the exclusions in each environment it would all Just Work but it appears not. Do 
select
rules at the environment level completely override those from the plugin?
e
Do 
select
rules at the environment level completely override those from the plugin?
Yes, and we should probably document it. The alternative of merging 
select
rules at different levels would quickly get messy. You could still accomplish something similar with YAML anchors if you have everything in 
meltano.yml
t
Thanks @edgar_ramirez_mondragon. Fair point about merging the rules being hard to get right (it depends on intent, I think, which is probably unknowable). I'm not familiar with YAML anchors, though from some quick googling I think I get it. Is your thought to put the "global" select rules in the plugin definition, with an anchor, and then reference them in the environment-specific rules via an alias?
I've been playing with this a little bit and I'm not having any luck for two reasons: 1. The 
select
item is a list; as far as I can anchors and aliases allow the value for an item to be overridden, but for a list doesn't that mean having to override the entire list? (Which defeats the purpose...) 2. The meltano.yml created using the CLI puts environments first and plugins second, so I can't reference anchors defined in plugins in environments without re-arranging the "top level" sections. Is it safe to assume meltano won't re-arrange the file if I do that and then make other config changes using the CLI? Or do I need to ensure that the file will only ever be changed by manual edits after that? Thanks!
Open in Slack
PreviousNext
Powered by