Catching the live stream for OO today and following the mapp Meltano #office-hours

Catching the live stream for OO today and followin...

ken_payne

08/02/2023, 4:53 PM

Catching the live stream for OO today and following the mappers discussion- especially @visch question about “where do you put custom code?”. Agree that a whole git repo for a custom mapper feels heavy - @pat_nadolny point - and the first thing that comes to mind is the Airflow DAGs folder paradigm. I.e. having a

mappers

folder full of

.py

files. Each file imports and subclasses a

BaseMapper

class, and we crawl the dir at the beginning of each run looking for the available custom mappers (similar to the Airflow DagBag concept). That way mappers are code, included in the Meltano Project and as lightweight as possible. On the backend we may still need to create a venv for the discovered mappers, and allow users to define requirements/dependencies for cases when a user wants to leverage 3rd party libraries/sdk (e.g. OpenAI 🤯 ) in custom mappers.

pat_nadolny

08/02/2023, 5:01 PM

I hadnt given it a ton of thought yet but this is similar to what I was thinking also. I like this idea!

pat_nadolny

08/02/2023, 5:07 PM

Another way to do it, like I was starting to get at in OH, is to make a generic mapper that doesnt do anything by default but it can be configured to reference a script (probably

Each file imports and subclasses a BaseMapper class

like you suggested for consistency) then they could create many distinct instance of that plugin using

inherit_from

pointing to different scripts, and potentially appending new dependencies directly to the pip_url although I dont think you can install the same plugin twice right now so it doesnt solve the separate venvs problem 🤔.

pat_nadolny

08/02/2023, 5:23 PM

Heres some sudo config of what I'm trying to describe. Basically 3 custom scripts that can be run inline, and one of them needs a custom nltk dependency so its added as an inherited plugin. I tried to test and it seems like it does create a new venv for the inherited child, maybe because it detected a different pip_url? idk

Copy code

- name: generic-mapper
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/generic-mapper.git>
    mappings:
    - name: clean-html
      config:
        code_path: $MELTANO_PROJECT_ROOT/mappers/generic-mapper/clean-html.py
    - name: add-embeddings
      config:
        code_path: $MELTANO_PROJECT_ROOT/mappers/generic-mapper/add-embeddings.py

  - name: generic-mapper-child
    inherit_from: generic-mapper
    pip_url: git+<https://github.com/MeltanoLabs/generic-mapper.git> nltk
    mappings:
    - name: nltk
      config:
        code_path: $MELTANO_PROJECT_ROOT/mappers/generic-mapper/nltk-clean.py

Open in Slack

Previous Next