#3202 Allow mappings to be "locally scoped" to specific extractors and loaders Issue created by MeltyBot Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3277 Originally created by @aaronsteers on 2022-02-22 225313 --- Currently all mappings are globally scoped. Meaning, if we want to define a

remove-pii

mapping for both

tap-shopify

and

tap-gitlab

, then we'll need to define them in a way to avoid name collision: probably

remove-shopify-pii

and

remove-gitlab-pii

. Click to expand example: defining mappers and mappings globally mappers: - name: transform-field namespace: pipelinewise_transform_field pip_url: pipelinewise-transform-field executable: transform-field mappings: - name: hide-gitlab-secrets config: transformations: - field_id: "author_email" tap_stream_name: "commits" type: "MASK-HIDDEN" - field_id: "committer_email" tap_stream_name: "commits" type: "MASK-HIDDEN" - name: fix-names config: transformations: - field_id: "given_name" tap_stream_name: "users" type: "lowercase" • If a user creates a mapping called

remove-pii

but accidentally applies it to the wrong tap, then it likely will fail or (worse) not filter or change any records at all, causing incorrect/prohibited data to land in the target. • If the system detects two mappings are both called

remove-pii

, then the proper behavior is to fail before executing so that the user can resolve the naming conflict. Context-aware application of the correct mapping Our initial long-term goal was to support locally-scoped mappings to that

remove-pii

could be declared locally as a mapping for

tap-shopify

and

tap-gitlab

(for instance), and each would be defined according to the needs of that extractor. 'Optional' vs 'Always On' Mappings With locally-scoped mappings, some mappings could be defined as 'always on' for a given extractor and others as 'optional', depending upon settings set within the mapping config. For example,

remove-pii

could be 'Always on' and

ten-percent-sample

or

southwest-region-only

could be 'optional', and applied only when explicitly requested. Side note: As a best practice, it probably makes sense to tie mappings to specific environments, so that each environment always uses the same mappings. Avoiding accidental mismatch In this case, if a third

tap-github

extractor is invoked with

remove-pii

, but there's no locally-defined definition, Meltano will know to abort and fail, rather than use a mapper scoped/configured for another extractor. Sample definitions inline with the tap/target Because locally-scoped mappings are specific to a single tap or target, their definition is most intuitively configured alongside the tap or target definition, as child to it. Hence, our original spec from #2299: Click to expand sample code mappers: - name: meltano-map-transform pip_url: meltano-map-transform - name: pipelinewise-transform-field pip_url: pipelinewise-transform-field - name: no-activate-version. # A fictional mapper that removes ACTIVATE_VERSION messages pip_url: no-activate-version extractors: - name: tap-gitlab # ... mappings: - name: pii-hasher # The name of the map transform to apply. mapper: meltano-map-transform # The mapper plugin to use. config: # What will be sent to the transformer in a config.json file. stream_maps: customers: id_hashed: md5(record['id']) id: None loaders: - name: target-salesforce # ... mappings: - name: add-global-guid # The name of the map transform to apply. mapper: meltano-map-transform # The mapper plugin to use. config: # What will be sent to the transformer in a config.json file. stream_maps: customers: guid: md5(record['id']) - name: target-csv # ... mappings: - name: flatten-records # The name of the map transform to apply. mapper: meltano-map-transform # The mapper plugin to use. default: true # Transform prepended automatically if default=true. config: # What will be sent to the transformer in a config.json file. flatten_records: true - name: compat-fix # The name of the map transform to apply. mapper: no-activate-version # The mapper plugin to use. # config is omitted if the generic config is sufficient In this example, the

mappings

collection can exist under any tap or target, and each mapping has: 1.

name

(what to call the mapping when invoked) 2.

default

(true or false whether this is applied by default) 3.

config

(whatever settings values are needed to configure the specific mapping) 4.

mapper

(a pointer to the plugin name that drives the transformation) meltano/meltano