https://linen.dev logo
Join Slack
Powered by
Hi, how’s it going? What is the correct way to pas...
# troubleshooting
f
Hi, how’s it going? What is the correct way to pass a setting 
kind: object
with environment variables to a plugin?
p
@flavio_siqueira_prado check out the environment variables docs https://docs.meltano.com/guide/configuration#environment-variables for more details but something like this should work 
TAP_NAME_SETTING_NAME='{"key": "value"}'
(obviously substituting your names in). You can run 
meltano config my-tap
to make sure the config is populating as expected
f
Thanks. Does it have any restriction? I had a json like 
TAP_NAME_SETTING_NAME={"special-key.*": "value"}
and I didn’t get it on the configuration file of the plugin.
p
hmm I didnt think so but the dot looks to be causing it to be excluded
@edgar_ramirez_mondragon @Will Da Silva (Arch) is this a known/expected result?
e
hmm I didnt think so but the dot looks to be causing it to be excluded
Even with wrapping single quotes?
p
Heres my test case:
Copy code
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key": "value"}' meltano config tap-google-analytics
{
  "client_secrets": {
    "special-key": "value"
  }
}
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key*": "value"}' meltano config tap-google-analytics
{
  "client_secrets": {
    "special-key*": "value"
  }
}
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key.*": "value"}' meltano config tap-google-analytics
{
  "client_secrets": {}
}
e
Yeah, I can confirm this as a bug. I think what’s going on is Meltano is detecting the dot as a delimiter and splitting it into 
special-key
and 
*
.
w
I suppose the question then becomes how can one use a config key with a 
.
in it with Meltano? We always expand such keys into nested dictionaries in the config.
@aaronsteers @taylor Have either of you run into this problem before? Needing a 
.
in a Meltano config key?
It would be easy to support some escape hatch like making it so that 
\.
doesn't get expanded, but 
.
does.
f
You use python 
json.loads
, if you add 
\.
you get an error on it about the escaped 
\
w
Yeah, I suppose it'd have to be 
\\.
in the JSON/YAML
f
Copy code
>>> import json
>>> json.loads('{"key\\.": "value"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/flavio.s.prado/.pyenv/versions/3.10.0/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/flavio.s.prado/.pyenv/versions/3.10.0/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/flavio.s.prado/.pyenv/versions/3.10.0/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 6 (char 5)
w
Not a particularly clean or obvious workaround, and the problem itself is bound to be surprising.
Thanks for checking that @flavio_siqueira_prado. I guess we can't use a backslash to work around this problem.
t
I haven’t run into this myself - seems odd that the quotes don’t do anything to prevent it either
w
I was actually just wondering why this isn't a problem yesterday since I was working on what seems to be a bug with how we expand/flatten config on the 
.
for the manifest feature.
@pat_nadolny @edgar_ramirez_mondragon @flavio_siqueira_prado Would this be an acceptable outcome?
Copy code
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key.*": "value"}' meltano config tap-google-analytics
{
  "client_secrets": {
    "special-key": {
      "*": "value"
    }
  }
}
Or does the key actually need to be 
special-key.*
?
If the above is what you want, then you can work around this problem by expanding the dot yourself:
Copy code
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key": {"*": "value"}}' meltano config tap-google-analytics
I've logged the issue around the keys with periods in the JSON config provided by env vars being skipped: https://github.com/meltano/meltano/issues/7317
This is separate from the issue of there being no way (as far as I know) to have a config key with a 
.
in it that shouldn't be expanded.
a
Have either of you run into this problem before? Needing a 
.
in a Meltano config key?
Since Meltano settings names can be anything, and since 
.
is a delimiter between nested config nodes, I think I'm totally okay with not supporting 
'.'
in setting names. That said, if the problem is that we're trying to send a complex object to the setting and we want a passthrough ignoring dots and wildcards in the field's contents, then I can see why this from @pat_nadolny's tests above is a bug preventing that use case:
Copy code
TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS='{"special-key.*": "value"}' meltano config tap-google-analytics
{
  "client_secrets": {}
}
w
No 
.
in setting names is an okay resolution to that, I think. It should be documented as a limitation with no workarounds. It's a separate matter from the linked bug, which is about the key-value pair not being processed because of the dot.
a
Got it. Yeah, makes sense 👍
f
@Will Da Silva (Arch) let me explain what was my intent, I wanted to modify 
target-redshift
setting 
schema_mapping
to accept a regex as a key. I wanted to do it because I have an use case of aggregating more than one schema from my tap to the same schema on redshift. As the 
.
is not supported I changed my fork and my PR (on redshift target) to support unix file pattern matching for now. Btw, thank you very much guys for the great support here.
Eg. 
TARGET_REDSHIFT_SCHEMA_MAPPING='{"shard.*": {"target_schema": "shards"}}'
So all schemas at my origin that would match 
shard.*
would have it’s tables on redshift schema 
shards
. Right now I changed for 
shard*
, not as powerful as regex but good enough for my case.
Open in Slack
PreviousNext
Powered by