vincent_legendre
01/03/2023, 1:44 PMmeltano-map-transformer
to rename streams, since the extractor (https://github.com/transferwise/pipelinewise-tap-postgres) and the loader (https://github.com/adswerve/target-bigquery) do not provide stream-maps capability.
The following configuration works fine :
mappers:
- name: meltano-map-transformer
pip_url: "git+<https://github.com/MeltanoLabs/meltano-map-transform.git>"
executable: meltano-map-transform
mappings:
- name: alias-my-schema
config:
stream_maps:
my-schema-taxis:
__alias__: taxis
However, when trying to inherit from the transformer to allow partial modification, I end up with the following error :
Error: Ambiguous mapping name alias-my-schema, found multiple matches.
My Meltano config here is divided in two parts :
• meltano.yml config file
# meltano.yml
mappers:
- name: meltano-map-transformer
pip_url: "git+<https://github.com/MeltanoLabs/meltano-map-transform.git>"
executable: meltano-map-transform
• sub config file :
mappers:
- name: map-transform--my-schema
inherit_from: meltano-map-transformer
mappings:
- name: alias-my-schema
config:
stream_maps:
my-schema-taxis:
__alias__: taxis
Btw I noticed that a tap-postgres variant (https://hub.meltano.com/extractors/tap-postgres--meltanolabs/) and a target-bigquery variant (https://hub.meltano.com/loaders/target-bigquery--youcruit/) are based on Meltano SDK and offer stream-map capabilities.
Should we plan on using Meltano SDK-based plugins by default, and thus promote these variants to the "default" variant in the future ?
(IMO the efforts put into Meltano SDK should benefit major taps and targets, especially the default variants melty rainbow flame 🔥 )