Is there any way to get a stream map config to expand an env Meltano #troubleshooting

Is there any way to get a stream map config to exp...

quinn_batten

01/05/2023, 11:44 PM

Is there any way to get a stream map config to expand an env variable? I want to hash some columns and I dont want to store the seed in plaintext in my yaml file, I want to keep it in an env var. But I cant seem to find a way to actually get it to expand… https://sdk.meltano.com/en/latest/stream_maps.html#constructing-the-stream-maps-config-object (more deets in thread)

quinn_batten

01/05/2023, 11:45 PM

This example uses the literal actual string “${MAPPER_HASH_SEED}” as the seed, rather than the value of the MAPPER_HASH_SEED env var, which is not what I want lol

Copy code

mappers:
  - name: meltano-map-transformer
    pip_url: "git+<https://github.com/MeltanoLabs/meltano-map-transform.git>"
    executable: meltano-map-transform
    mappings:
    - name: redact_email__email_address
      config:
        stream_maps:
          redact_email__email_address: &redact_email__email_address
            email_name_hash: md5(config['hash_seed'] + email_address.split('@')[0])
          public-exampletable: *redact_email__email_address
        stream_map_config:
          hash_seed: "${MAPPER_HASH_SEED}"

quinn_batten

01/05/2023, 11:52 PM

Adding the variable to the environment doesn’t solve either. Here’s a snippet from higher up in my meltano.yml file, which should be injecting that env var into the plugins env if it wasn’t there already

Copy code

environments:
- name: dev
  env:
    MAPPER_HASH_SEED: $MAPPER_HASH_SEED

Will Da Silva (Arch)

01/05/2023, 11:58 PM

@quinn_batten This is not currently supported. Could you please open an issue for https://github.com/meltano/meltano/. I'm currently working towards adding environment variable support in many locations, and improving it where it currently exists. I can see about supporting it for stream map configs too.

quinn_batten

01/05/2023, 11:58 PM

Ah gotcha. Thanks Will! I’ll make an issue

quinn_batten

01/06/2023, 12:21 AM

https://github.com/meltano/meltano/issues/7151

aaronsteers

01/06/2023, 3:10 AM

thanks, @quinn_batten! Might be caused by this one: https://github.com/meltano/meltano/issues/3171 I've linked the two for easy cross-reference.

edgar_ramirez_mondragon

01/06/2023, 7:46 PM

@quinn_batten maybe the example in this issue is helpful? https://github.com/meltano/sdk/issues/1073

Open in Slack

Previous Next