I'm adding in the meltano transformer, but I'm see...
# plugins-general
d
I'm adding in the meltano transformer, but I'm seeing something odd when running
meltano install
. Here's the relevant config section:
Copy code
mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
      - name: test
        config:
          stream_maps:
             ...
This is the output of
meltano install
:
Copy code
2024-08-07T13:30:29.029274Z [info     ] Installing 4 plugins
2024-08-07T13:30:29.029917Z [info     ] Skipped installing mapper 'meltano-map-transformer'
2024-08-07T13:30:29.030503Z [info     ] Installing extractor 'tap-mssql'
2024-08-07T13:30:29.136900Z [info     ] Installing loader 'target-snowflake'
2024-08-07T13:30:29.162643Z [info     ] Installing mapper 'meltano-map-transformer'
2024-08-07T13:30:36.287505Z [info     ] Installed extractor 'tap-mssql'
2024-08-07T13:30:50.561328Z [info     ] Installed loader 'target-snowflake'
2024-08-07T13:30:51.726874Z [info     ] Installed mapper 'meltano-map-transformer'
2024-08-07T13:30:51.727671Z [info     ] Installed 3/4 plugins
2024-08-07T13:30:51.727821Z [info     ] Skipped installing 1/4 plugins
Somehow, the transformer is being picked up twice and skipped once. If I remove the
mappings
from the config and leave in the parent section with the name, variant, and url, I get the expected result:
Copy code
2024-08-07T13:29:19.894188Z [info     ] Installing 3 plugins
2024-08-07T13:29:19.895097Z [info     ] Installing extractor 'tap-mssql'
2024-08-07T13:29:20.048231Z [info     ] Installing loader 'target-snowflake'
2024-08-07T13:29:20.091293Z [info     ] Installing mapper 'meltano-map-transformer'
2024-08-07T13:29:27.236490Z [info     ] Installed extractor 'tap-mssql'
2024-08-07T13:29:36.914672Z [info     ] Installed loader 'target-snowflake'
2024-08-07T13:29:38.103011Z [info     ] Installed mapper 'meltano-map-transformer'
2024-08-07T13:29:38.103867Z [info     ] Installed 3/3 plugins
If I completely remove
mappers
, then it's down to 2 as expected. So my question is why does adding
mappings
duplicate it? Or did I set up my config incorrectly somehow?
r
I think Meltano is treating each
mapping
for a
mapper
as special kind of plugin internally, and skips installing them with a confusing info log as you have seen:
Copy code
2024-08-07T13:30:29.029917Z [info     ] Skipped installing mapper 'meltano-map-transformer'

...

2024-08-07T13:30:29.162643Z [info     ] Installing mapper 'meltano-map-transformer'

...

2024-08-07T13:30:51.726874Z [info     ] Installed mapper 'meltano-map-transformer'
Loosely related to https://github.com/meltano/meltano/issues/8660 and could be addressed by a common fix.
👍 1