mark_johnston
04/17/2023, 11:52 PMmeltano-map-transformer
- while most taps/targets have fairly simple equivalent environment variables for things like connection settings, meltano-map-transformer
only has one environment variable: MELTANO_MAP_TRANSFORMER_STREAM_MAPS
which doesn't seem enough to configure all the potential mappers that you may wish to use, it is also quite hard to discern a valid value for this environment variable. I.e. trying to build an equivalent for this meltano.yml
excerpt:
mappings:
- name: rename_jobs_to_dbt_jobs
config:
stream_maps:
jobs:
__alias__: dbt_jobs
Is it possible to use the MELTANO_MAP_TRANSFORMER_STREAM_MAPS
variable and what would be a valid value for it?pat_nadolny
04/18/2023, 2:42 PMpat_nadolny
04/18/2023, 2:43 PMpat_nadolny
04/18/2023, 2:44 PMmark_johnston
04/20/2023, 6:45 PMmeltano.yml
file is not being updated regularly, but we do all our configuration using environment variables from AWS SSM Parameter Store, or setup directly on ECS Tasks. This works really well for taps and targets, but not as well for mappers. I want to build a few meltano jobs with mappers that do different things depending on the tap/target and stream.