Hello, I have a use case where I need to convert ...
# troubleshooting
a
Hello, I have a use case where I need to convert some field values to string type before they are sent to the loader and I am doing it using the
meltano-map-transform
mapper. Its configuration doesn't seem to support wildcards when declaring the stream id. Am I missing something or is this currently not supported?
v
While a map-transform should work 100% of the time, I'd try to use https://docs.meltano.com/concepts/plugins#schema-extra if you can get away without having a mapper!
a
I actually tried to change the type using the schema extra, but it looks like it is ignored by the loader (I tried to dump the catalog.json file so I'm sure the change is applied properly). I have a field of type
['null', 'object']
and even after setting it to
['null', 'string']
the loader still complains because it is expecting an object. I'm still new to the singer/meltano ecosystem but to me it seems there's some problem with the mysql tap, and I opened a dedicated issue.
v
Got it! Yeah I'm not super well versed in the mapper stuff yet in the sdk. I don't think there's a wild card for all stream names but I don't know!
a
@alessandro_mangone AFAIK you have to explicitly specify each stream and field by name at the moment. I was hoping to do something similar (in my case, convert CamelCase to snake_case). I'll write up a feature request ticket.
a
Yes please! I think this would be quite helpful in the case you have a mysql tap with tens of schemas all containing the same tables and you want to apply the transformation to all of them, or in general any case where you have lots of similar streams