Hello Team, I'm using meltano latest version (1.80...
# troubleshooting
f
Hello Team, I'm using meltano latest version (1.80.1) and found environment variables expansion not worked in
meltano.yml
plugins
extras
. Does it only work in plugins
config
now?
d
@francis_niu It should also work in extras, can you show me the
meltano.yml
snippet that isn’t working as expected?
f
My config files are like this:
Copy code
# .env
TAP_MONGODB_DATABASE=my_db_name

# meltano.yml
plugins:
  extractors:
  - name: tap-mongodb
    variant: singer-io
    pip_url: tap-mongodb
    select:
    - ${TAP_MONGODB_DATABASE}-user.*
    metadata:
      '*':
        replication-method: LOG_BASED
    schema:
      ${TAP_MONGODB_DATABASE}-user:
        _id:
          type: string
        name:
          type: string
Then I run
meltano select tap-mongodb --list --all
, it outputs as below and cannot select expected collections:
Copy code
Enabled patterns:
    ${TAP_MONGODB_DATABASE}-user.*
Additional information: Variable expansion in
metadata
extra works fine, like this:
Copy code
metadata:
      '*':
        replication-method: $REPLICATION_METHOD
But in
select
and
schema
extras doesn’t work. My current workaround is putting all
select
into
.env
and use wildcard to replace db name in
schema
(because schema is too long for env var). It’s weird but works fine.
d
@francis_niu You’re running into a known issue, unfortunately: https://gitlab.com/meltano/meltano/-/issues/2481 Env var substitution also doesn’t work in keys (like your
${TAP_MONGODB_DATABASE}-user
under
schema
), just in values.
@taylor May be worth a bump in priority?
@francis_niu Since the values in
select
and the keys in
schema
are patterns, would
*-user.*
work, or would that pick up unrelated
*-user
tables?
f
Yes, it worked. Thanks! I put
select
into .env to specify tables precisely and use wildcard pattern in
schema
.