Hi all, has anyone had the need to mask production...
# plugins-generala
aaron_phethean
09/27/2022, 4:50 PMHi all, has anyone had the need to mask production data with Meltano yet? Seems like a pretty common use case - maybe there are some good ideas here. As a first thought, I'm wondering if it would be generally useful to create a mapper plugin that uses faker (https://faker.readthedocs.io/en/master/). That mapper could deal with the PII or client account names - I've explored Mostly.ai but didn't have much joy yet to be honest.
v
visch
09/27/2022, 4:53 PMFor all my use cases for production PII I just don't pull the PII data, either use a mapper to filter it or just tell the source to stay away from it
a
aaronsteers
09/27/2022, 5:22 PMI haven't had the personal application opportunity, but I definitely agree with mappers for that use case. (That's probably the main use case for them, actually.)
aaronsteers
09/27/2022, 5:24 PMI thought about Faker and similar libraries for tap-smoke-test, but not for the mapper class. I actually really like the idea though and I think it could be worth adding a built-in
faker()md5()aaronsteers
09/27/2022, 5:26 PMThere's a counter argument about dependencies, but this one doesn't seem to have much in the way of additional dependency requirements: https://github.com/joke2k/faker/blob/master/setup.py#L73-L77
e
edgar_ramirez_mondragon
09/27/2022, 5:27 PMThose two deps are already in the sdk 🙂
a
aaronsteers
09/27/2022, 5:27 PMAs of today, you should already be able to use a hashing function like
md5()aaronsteers
09/27/2022, 5:29 PM@aaron_phethean - would you be interested in logging an issue for this and/or perhaps contributing the feature?
If added to the SDK, you also would automatically get access to that in taps and targets that use the SDK - even without having a standalone mapper between them.
a
aaron_phethean
09/27/2022, 6:37 PM100% will do that first thing in the morning. Maybe PR by the afternoon 😁