https://linen.dev logo
Join Slack
Powered by
hi all, I'm trying to understand how to leverage m...
# getting-started
j
hi all, I'm trying to understand how to leverage mappers to create an alias for a table. it's mentioned in the docs and in this Slack that one has to run 
meltano run tap mapper target
, does this effectively replace 
meltano elt tap target --job-id=job-id
? I don't see a 
--job-id
in 
meltano run
, so I'm not sure how the former tracks state. in addition, I'm getting a 
BrokenPipeError
when I try to run my mapper, details in thread 👇
when I try 
meltano invoke meltano-map-transformer --about
, I get
Copy code
2022-05-05T11:51:35.438136Z [info     ] Environment 'dev' is active
time=2022-05-05 11:51:36 name=meltano-map-transformer level=INFO message=meltano-map-transformer v[could not be detected], Meltano SDK v0.4.2)
Traceback (most recent call last):
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/bin/meltano-map-transform", line 8, in <module>
    sys.exit(StreamTransform.cli())
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/singer_sdk/mapper_base.py", line 140, in cli
    config_dict = merge_config_sources(
  File "/project-dir/meltano/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/singer_sdk/configuration/_dict_config.py", line 83, in merge_config_sources
    config.update(read_json_file(config_path))
TypeError: 'NoneType' object is not iterable
and looks like the same error appears when I do 
meltano elt tap mapper target
. this is my relevant config:
Copy code
plugins:
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
environments:
- name: dev
  config:
    plugins:
      mappings:
      - name: alias-tables
        config:
          stream_maps:
            app_users:
              __alias__: orchest_cloud_users
            app_instances:
              __alias__: orchest_cloud_instances
well, I saw that there are 
stream_maps
supported without the need to use mappers, but despite https://gitlab.com/meltano/sdk/-/issues/301 appearing as fixed, I can't make them work. will keep trying later 😉
t
I believe stream_maps only works if the tap you're working with is based on the Meltano SDK... otherwise you need the mapper
meltano run
does replace 
meltano elt
. You don't have to specify a job_id with the 'run' command because Meltano creates it for you in that case.
Does the pipeline actually not work? It sounds like the error is only when you use 
meltano invoke
?
Confirmed that the 
--about
switch appears to be broken though 🤨
j
the pipeline fails with the same error but the traceback is interleaved with the log messages and it's a bit difficult to digest, so I pasted the cleaner one coming from 
--about
I believe stream_maps only works if the tap you're working with is based on the Meltano SDK...
I see, that makes sense (because I didn't see any mention of it in the Singer spec)
doing some long debugging I extracted that apparently I was missing a required 
mapping_name
extra config. https://gitlab.com/meltano/meltano/-/merge_requests/2481 introduced the whole code and hasn't been touched since. will update the bug
bad news, 
--about
got fixed but 
meltano run
now reports 
Error: Ambiguous mapping name alias-tables, found multiple matches.
. giving up for today, will revisit this in a few days
well, I couldn't help myself 😉 what I recollected: • despite 
meltano invoke meltano-map-transformer --about
being broken, 
meltano run tap mapper-name target
works 👍 couldn't reproduce the 
BrokenPipeError
anymore, probably I had a bad config somewhere • 
__alias__
is also working well, but I was using the wrong table name (say, 
users
instead of 
public-users
)
t
@juan_luis_cano_rodriguez can you add your comments to https://gitlab.com/meltano/meltano/-/issues/3309 to help us track?
j
sure thing @taylor 👍 will add one soon
a
Hi @juan_luis_cano_rodriguez! Thanks for sharing, I’m having the same issue with 
meltano-map-transformer
: 
TypeError: 'NoneType' object is not iterable
, both when I try to run it as part of a pipeline (
meltano run tap meltano-map-transform target
) or with 
meltano invoke meltano-map-transform --about
What changes did you apply to your config to fix the issue when running the pipeline?
j
hi @alex! long story short: • 
meltano invoke meltano-map-transform --about
is broken, don't rely on it • 
meltano run tap meltano-map-transform target
should work if properly configured, this is how the relevant section of my 
meltano.yml
looks like:
Copy code
plugins:
  extractors: ...
  loaders: ...
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: alias-tables
      config:
        stream_maps:
          public-app_users:
            __alias__: cloud_users
          public-app_instances:
            __alias__: cloud_instances
          public-posthog_event:
            __alias__: telemetry_events
          __else__: null
a
Thanks, indeed, this was mostly a config issue! I still get some weird behavior, for instance when trying to drop some columns (with 
column_name: null
), but at least it’s working now (using 
__else__: null
)
Open in Slack
PreviousNext
Powered by