I made a custom mapper (a very small modification to meltanolabs’

meltano-map-transform

), and while it wasn’t difficult, it still made me wonder how it could be easier to add a custom transformation. In my solution, I had to subclass two classes from meltano_sdk: Both

PluginMapper

and

CustomStreamMap

, because the additional feature I wanted to modify was in

CustomStreamMap

, but that class is instantiated in

PluginMapper

. Basically, I posit that common modifications to the map transform is in modifying the

CustomStreamMap

class, and it makes sense to make that class more accessible. I have doodled a possible way to do this by letting

PluginMapper

accept a

mapper_class

argument that defaults to the SDKs

CustomStreamMap

, but can be changed without altering

PluginMapper

. In other words, anyone who wants to alter the mapper can fork it, create a subclass of

CustomStreamMap

with the added functionality and pass it in to

PluginMapper

. The code: https://github.com/radbrt/meltano-map-transform/tree/pluggable_mapper Any thoughts are welcome.

I just had a quick read-through. I think something like the plugin_mapper class property you suggest makes a lot of sense! It's similar to the SQLTap class using a default stream class that can be overridden and similar also to Targets having a default_sink_class.

The difficulty of doing stuff like this makes me really wish that meltano worked more like airflow (just being a python library with classes you can subclass/modify as part of the meltano process) rather than its current setup where you either write no code or you create a whole separately installed python package

I see what you mean, there are a lot of connectors that are just some other connector with a minor tweak. But I am increasingly OK with maintaining modified versions of these repos, the overhead ins’t that big and it keeps me from mangling the logical separation of concerns. Early on I tried to install two taps/targets in the same venv, and I quickly learned why we don’t do that. If that problem could somehow be solved, I would be interested in exploring some python-programming version of Meltano where the data was passed as dicts between tap and target classes. That would be… different.

Agreed with your comments, @Henning Holgersen. I don't know that we'd ever want to completely replace/remove the virtualenv installation approach, but one thing we did consider was to allow arbitrary libraries/modules to be referenced from the mapper itself. That has some security implications, to be sure, but if we wanted to, we could allow arbitrary imports via config, including: • Standard libraries and functions (like

os

or

os.environ

). • Custom libraries that would presumably need to be appended to the Pip URL. • Local paths/modules with custom code. The third option could get us close to something like your proposal, @ptd. But mappers are somewhat unique in being able to greatly benefit from arbitrary grafts from local code or other already-existing modules. Taps and targets generally require the dedicated repo/library install source, but I guess even those could use local project-relative install sources if/when it's preferable to keep code in the project repo... Wdyt?

If we focus on the mapping-question, I do think we are in a kind of penumbra where we are introducing fairly arbitrary programming possibilities into what is otherwise standardised components. I’d be able to argue that all the tweaks to taps/targets demonstrate a lack of investment in good common ones, but with mappers the entire point is to introduce anything that can be expressed in code. From a usability perspective though, perhaps the best solution would be if I could maintain a library consisting only of the transform functions, and reference it from the config. This example got clumsy, but you get my point:

mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/meltanolabs/meltano-map-transform.git>
    mappings:
    - name: some-transform
      config:
        stream_maps:
          some-stream:
            _batch_id: 
              library: git+<https://github.com/radbrt/hennings-transforms>
              method: my_super_transform
              input: record["my-untransformed-column"]

Yes, this to me seems very similar to what I was imagining in the "python_imports" mock-up here: https://github.com/MeltanoLabs/meltano-map-transform/issues/11

Indeed, can and should are two very different things, my phrase “from a usability perspective” above does a lot of heavy lifting. In practice, I think we have other things to spend our time on. The small proposed change above would be a great start, and if we have time, perhaps a cookiecutter template so that it’s easier not to eternally fork the meltano-map-transform library might be a pragmatic approach.

Hi @Henning Holgersen @aaronsteers I am trying to do something similar. I have a use case where I want to flatten an Array of Object into individual

RECORDS

. Its like exploding the array. I am going through the code, I have identified

transform

method in

CustomStreamMap

to modify. Your thoughts are welcome.