Hello all, looking for some assistance using plugin env vars in a meltano.yml loader config:

plugins:
  loaders:
    - name: ga-target-s3-jsonl
      inherit_from: target-s3-jsonl
      env:
        PREFIX: the_prefix_here
      config:
        naming_convention: meltano/${PREFIX}/{stream}/export_date={date:%Y-%m-%d}/{timestamp}-{stream}.json

but when I run

meltano config --format=env target-s3-ga-daily-campaign-performance list

the PREFIX env is not string interpolated to the s3 path:

'meltano-test/datalake-json//{stream}/export_date={date:%Y-%m-%d}/{timestamp}-{stream}.json'

and when I run

meltano elt tap-google-ads ga-target-s3

the env is not added to the expected config naming_convention string. Any ideas why the env var is not being interpolated correctly?

Not 100% sure, but I don't think environment variable interpolation is supported in

meltano.yml

currently. The mapping you defined in

env

just means that

PREFIX

will be available to

ga-target-s3-jsonl

at runtime - not that meltano will understand to interpolate the

naming_convention

value with it. This PR (specifically, this part) and this issue are exploring env var interpolation support, but for specific features only. I'm unsure of the implication of enabling this behaviour

meltano.yml

-wide... Either way, I would open an issue for your use-case. See below.

Hi, I think my problem is related with yours. I used DBT_BIGQUERY_PROJECT env variable in way like this but now seems it doesn’t work anymore

- name: dbt-bigquery
    config:
      dataset: dbt_stage
    env:
      DBT_PACKAGE_GA4__PROJECT: ${DBT_BIGQUERY_PROJECT}

@ken_payne In reference to your comment, I think my answer to this is incorrect. Would you mind clarifying how environment variables are interpolated in

meltano.yml

(what is/isn't supported)?

Hey 👋 Here is what is currently supported (and what we have tests for). The code that constructs the environment to use for expansion is here (during

get_with_metadata

) and all values read via the

MeltanoYmlStoreManager

are marked as

expandable

here. All that to say, what you both are doing is/should be supported 🤔 I have also just tried your example @robert_espinoza in a fresh project and expansion is working as expected:

Hi @ken_payne thanks for the follow up. I am running meltano version

2.1.0

. Do you know what versions support expansion?

No problem, glad to help. It's hard to say. We have had expansion for some parts of the hierarchy all along, but only more recently (

2.4.0

) for other parts (specifically the plugin-level env you are using). It may be that by moving your env to a point higher in the expansion hierarchy, your expansion will work in version

2.1.0

, but I would always suggest upgrading to the latest version to make use of fixes and new features 🙂 E.g. using the environment-level env key, in stead of the plugin level:

environments:
- name: dev
  env:
    PREFIX: the_prefix_here
plugins:
  loaders:
    - name: ga-target-s3-jsonl
      inherit_from: target-s3-jsonl
      config:
        naming_convention: meltano/${PREFIX}/{stream}/export_date={date:%Y-%m-%d}/{timestamp}-{stream}.json

As far as I remember, we added expansion at the

environment

env level at the same time as the environments feature, so this will likely work 😅

Thanks @ken_payne 🙂