Daniel Luo
08/07/2024, 1:38 PMmeltano install
. Here's the relevant config section:
mappers:
- name: meltano-map-transformer
variant: meltano
pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
mappings:
- name: test
config:
stream_maps:
...
This is the output of meltano install
:
2024-08-07T13:30:29.029274Z [info ] Installing 4 plugins
2024-08-07T13:30:29.029917Z [info ] Skipped installing mapper 'meltano-map-transformer'
2024-08-07T13:30:29.030503Z [info ] Installing extractor 'tap-mssql'
2024-08-07T13:30:29.136900Z [info ] Installing loader 'target-snowflake'
2024-08-07T13:30:29.162643Z [info ] Installing mapper 'meltano-map-transformer'
2024-08-07T13:30:36.287505Z [info ] Installed extractor 'tap-mssql'
2024-08-07T13:30:50.561328Z [info ] Installed loader 'target-snowflake'
2024-08-07T13:30:51.726874Z [info ] Installed mapper 'meltano-map-transformer'
2024-08-07T13:30:51.727671Z [info ] Installed 3/4 plugins
2024-08-07T13:30:51.727821Z [info ] Skipped installing 1/4 plugins
Somehow, the transformer is being picked up twice and skipped once. If I remove the mappings
from the config and leave in the parent section with the name, variant, and url, I get the expected result:
2024-08-07T13:29:19.894188Z [info ] Installing 3 plugins
2024-08-07T13:29:19.895097Z [info ] Installing extractor 'tap-mssql'
2024-08-07T13:29:20.048231Z [info ] Installing loader 'target-snowflake'
2024-08-07T13:29:20.091293Z [info ] Installing mapper 'meltano-map-transformer'
2024-08-07T13:29:27.236490Z [info ] Installed extractor 'tap-mssql'
2024-08-07T13:29:36.914672Z [info ] Installed loader 'target-snowflake'
2024-08-07T13:29:38.103011Z [info ] Installed mapper 'meltano-map-transformer'
2024-08-07T13:29:38.103867Z [info ] Installed 3/3 plugins
If I completely remove mappers
, then it's down to 2 as expected. So my question is why does adding mappings
duplicate it? Or did I set up my config incorrectly somehow?Reuben (Matatika)
08/07/2024, 3:16 PMmapping
for a mapper
as special kind of plugin internally, and skips installing them with a confusing info log as you have seen:
2024-08-07T13:30:29.029917Z [info ] Skipped installing mapper 'meltano-map-transformer'
...
2024-08-07T13:30:29.162643Z [info ] Installing mapper 'meltano-map-transformer'
...
2024-08-07T13:30:51.726874Z [info ] Installed mapper 'meltano-map-transformer'
Loosely related to https://github.com/meltano/meltano/issues/8660 and could be addressed by a common fix.