Is there any way to configure a stream map so that...
# getting-started
h
Is there any way to configure a stream map so that all table/stream names are converted to lower case? I'm currently having to write a map for every table explicitly.
r
I don't see any docs, but it should be possible with https://github.com/meltano/sdk/pull/1888, which was added in SDK
v0.37.0
(fairly recent). If you are configuring
stream_maps
on a tap/target, it will need to be built with that SDK version or later - otherwise you can use meltano-map-transformer.
h
That is good to know, although it's how clear or written how to achieve what I'm after.
Copy code
mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: test
      config:
        stream_maps:
          dbo-*:
            __alias__: self.lower()
This will write the literal "self.lower()" out as the stream name, I can't seem to write an expression.
e
It's not currently supported, but see https://github.com/meltano/sdk/commit/541d2ae12974652462ebcb1c5275c799b9d81e6d. I'm not sure whether I'll be able to dedicate more time to it, but it might be a good starting point if you're interested in contributing 🙂
Your requirement is easier than to implement than where I was trying to get in that WIP branch, which made the generated schemas dependent on incoming RECORD messages.
âś… 1
h
That's great information, thank you. We will very likely need to modify the source code, and create custom connectors, which we can create pull requests for. This won't be immediate however, as we focus on getting our product up and running and it will take some time to understand the source code. For now, we'll write a script to generate the necessary entries in the config.
👍 1
e
Gotcha. Post here in Slack if you get stuck.