It would be nice if meltano is able to do redactio...
# randomh
haleemur_ali
10/18/2021, 9:34 PMIt would be nice if meltano is able to do redactions and masking. That would let me use it in settings where sensitive information cannot end up in the target system
d
dan_ladd
10/18/2021, 9:55 PMFor sdk based taps, you can do this with stream_maps. The biggest limitation though is it doesn't work for any nested fields.
dan_ladd
10/18/2021, 9:56 PMI know there was discussion of having a similar meltano transform feature sit between the tap and target for non sdk taps, but I'm not sure the status on it
a
aaronsteers
10/18/2021, 10:03 PMComing soon to Meltano also melty rainbow flame
We're doing some refactoring right now that will help enable this, but the vision is that Meltano pipelines will be able to inject a mapper plugin between the tap and target.
aaronsteers
10/18/2021, 10:04 PMaaronsteers
10/18/2021, 10:06 PMh
haleemur_ali
10/19/2021, 8:20 AMsimpleeval looks cool!the internal etl that I'm hoping meltano could replace uses https://github.com/jmespath/jmespath.py for the mapping. we can define custom mapping functions, as well as specify the message transform in a query-like language