Has anybody an idea if I can define a stream mapper mappings across multiple

meltano.x.yml

files? Basically I want to co-locate the mappings with their source, so I have meltano.foo.yml

plugins:
  mappers:
  - name: meltano-map-transformer
    mappings: 
    - name: FOO

and then another meltano.bar.yml file with

plugins:
  mappers:
  - name: meltano-map-transformer
    mappings: 
    - name: BAR

It seems I can't figure this out as meltano complains I redefine the meltano-map-transformer. But i can also not figure out how to add 2 independent instances of the transfomer either.

Maybe you can create two different mappers specific to BAR and FOO, that

inherit_from: meltano-map-transformer

how'd i do that without running into name conflicts? e.g. can i give them two different `name`s I'm asking because when composing job tasks it seem i only call

$loader $mapping $target

and never declare which mapper the mapping is from?

yes you can give them two different names. As far as you inherit from meltano-map-transformer (that can be empty)

I've not tried this before personally but in theory, @umar_saleem's proposal could work. You'd have a few different mappers, with inherit_from linking the bespoke ones back to the "main" declaration, and then you should be able to declare mappings in the mapper declaration that is adjacent to (aka in the same file as) your tap or target.

Hi All - my team has recently had this same question. We tried to implement the above suggested solution (inheriting from a base mapper declaration) and have had some success, but also ran into one failure scenario. Sorry for the extremely long message, but I felt it was important to lay out what all we've tried! First, the success: We can confirm that you can declare an explicit inheritance from a main declaration. The test scenario below is for two different taps (to rule out a specific tap issue) named

dayforce

and

freshservice

. The main declaration yml file:

plugins:
  mappers:
  - name: meltano-map-transformer
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git@v0.2.0>
    mappings:
    - name: dayforce
      config: 
        stream_maps:
          employees:
            __alias__: ZKEYS_EMPLOYEES
            XRefCode: XRefCode
            __else__: null

The child:

plugins:
  mappers:
  - name: meltano-map-transformer--new
    inherit_from: meltano-map-transformer
    mappings:
    - name: freshservice
      config: 
        stream_maps:
          tickets:
            __alias__: ZKEYS_TICKETS
            id: id

The above configuration works, but is not what we're actually trying to achieve. Second, the failure: We are trying to drop all fields except the

id

field, so we added an

__else__: null

to the

stream_maps

configuration in the child. Then, the child configuration looks like this:

plugins:
  mappers:
  - name: meltano-map-transformer--new
    inherit_from: meltano-map-transformer
    mappings:
    - name: freshservice
      config: 
        stream_maps:
          tickets:
            __alias__: ZKEYS_TICKETS
            id: id
            __else__: null

When we run the meltano job with the second configuration, it fails with a

KeyError: 'stream_maps.tickets.__else__'

error. I've attached a log file from a recent run with the error. If we move the whole 'freshservice' configuration back to one file and remove the inheritence, the same configuration works. It looks like this:

plugins:
  mappers:
  - name: meltano-map-transformer
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git@v0.2.0>
    mappings:
    - name: dayforce
      config: 
        stream_maps:
          employees:
            __alias__: ZKEYS_EMPLOYEES
            XRefCode: XRefCode
            __else__: null
    - name: freshservice
      config: 
        stream_maps:
          tickets:
            __alias__: ZKEYS_TICKETS
            id: id
            __else__: null

TL;DR: Explicit inheritance works! Except if we try to declare an

__else__

mapping.

@scott_goering - Can you try with

__else__: 'NULL'

- sending a string text "NULL" instead of an actual null value?

Thanks AJ. Will send it over to the team and let you know what the results are!

Hi @scott_goering! That should be fixed by https://github.com/MeltanoLabs/meltano-map-transform/pull/104, to be included tomorrow in a patch release 🙂

Thanks @edgar_ramirez_mondragon! I did a re-install of the mapper plugin in the test project, updated the mapper config to use

__NULL__

, and ran the job – and it ran successfully! I sent the commit link and info to the rest of the team to try in a 'real' project. I think we are good to go! Thanks for the super fast response. This is just one of the reasons why I love the Meltano team and community!