https://linen.dev logo
Join Slack
Powered by
With `meltano-map-transformer` is it possible to w...
# plugins-general
a
With 
meltano-map-transformer
is it possible to wildcard the actual field (say if you want to filter any record that has an illegal unicode character that would make the loader barf)?
a
No, not as of today. You can do wildcard conditions in row filters but but that would filter the rows themselves, and I don't know if there'd be any way to filter "any property".
Happy to accept an issue though if you don't mind logging one for us in the SDK project.
a
Happy to, but let me make sure what Im trying to do actually makes sense. Still trying to wrap my head around mappers.
Are
Copy code
transform-field
meltano-map-transformer
basically the same thing? Sounded like it from some threads here on slack.
a
They are solving similar problems but with a completely different set of config options and capabilities. The first is from Pipelinewise and actually has been around for a little while, although only recently supported within Meltano. The second is inspired by the first but authored by us (Meltano).
Docs for each...
Your use case description actually could be a good application - just that there's no config option for "catch/update X in any field". In theory, the 
__else__
option in the Meltano mapper could be expanded to perform this function, but as of now only 
__else__ = null
is supported, which has the affect of removing anything not otherwise declared.
a
Right, got it. I think I just misread something earlier about the two.
a
No worries. Glad you asked. 👍
a
Yeah, something like
Copy code
stream_maps:
  '*':
     __filter__: ...
does feel a bit like 
select
wildcards.
I haven’t found any example configs of the standalone 
meltano-map-transformer
, but if I am understanding correctly then the “tap level stream_maps” (ie, https://gitlab.com/meltano/squared/-/blob/master/data/meltano.yml#L28 ) should translate to something like…
Copy code
mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: filter-unicode
      stream_maps:
        __filter__: ...
(maybe mappings.config.stream_maps)
2 Views
Open in Slack
PreviousNext
Powered by