Hi team, how can we process data before loading to...
# best-practices
l
Hi team, how can we process data before loading to target? Currently the transformer runs after data loaded into target, but we want to hash certain columns of a table before loading to target, for example the column contains PII data. What is the best way to do it?
e
hi @lidong_zhao, mapper plugins might be what you're looking for 😄
l
Great, thanks.
I tried to run this command :
meltano run tap-upmssql transform-field target-snowflake
but the mapper config generated is empty. here is the content of plugins->mappers section, did I miss anything?
plugins:
mappers:
- name: transform-field
variant: transferwise
pip_url: pipelinewise-transform-field
executable: transform-field
mappings:
- name: transform-field
config:
transformations:
- field_id: Name
tap_stream_name: dbo-Test
type: MASK-HIDDEN
I debugged this issue, it turns out that a property mapping_name has to added to the mappers extras, and its value need to match with the name under mappings. It doesn't seem been documented anywhere.
e
@lidong_zhao thanks for debugging and sharing the solution. Would you mind filing an issue to document it? https://github.com/meltano/meltano/issues/new
l