Starting out trying to use `meltano-map-transforme...
# plugins-general
m
Starting out trying to use
meltano-map-transformer
- while most taps/targets have fairly simple equivalent environment variables for things like connection settings,
meltano-map-transformer
only has one environment variable:
MELTANO_MAP_TRANSFORMER_STREAM_MAPS
which doesn't seem enough to configure all the potential mappers that you may wish to use, it is also quite hard to discern a valid value for this environment variable. I.e. trying to build an equivalent for this
meltano.yml
excerpt:
Copy code
mappings:
    - name: rename_jobs_to_dbt_jobs
      config:
        stream_maps:
          jobs:
            __alias__: dbt_jobs
Is it possible to use the
MELTANO_MAP_TRANSFORMER_STREAM_MAPS
variable and what would be a valid value for it?
p
@mark_johnston this is a good question. I also gave it my best shot and wasn't able to get anything working. This probably deserves an issue. Can you talk more about your use case for why you want to set them using env vars vs directly in the meltano.yml?
Oh actually I found https://github.com/meltano/meltano/issues/6811 that was already opened
Heres the slack thread where that issue came from https://meltano.slack.com/archives/C01TCRBBJD7/p1664322242831029
m
Thank you @pat_nadolny - my use case is similar to @chris_evans requirement. We run Meltano in containers where the
meltano.yml
file is not being updated regularly, but we do all our configuration using environment variables from AWS SSM Parameter Store, or setup directly on ECS Tasks. This works really well for taps and targets, but not as well for mappers. I want to build a few meltano jobs with mappers that do different things depending on the tap/target and stream.