We would like to contribute an integration between Meltano and https://jsonata.org. WDYT? Adding @douwe_maan for visibility

@alberto_miorin interesting! I'd never heard of jsonata before, seems useful. Contributions are always welcomed! Can you share more about how you use it along side Meltano? How were you imagining an integration would look?

We would like a mapper on steroids. 🙂

Thats and interesting idea! Have you tried implementing it inside a mapper plugin already?

@alberto_miorin - Thanks for sharing this. I agree see the potential here for a very nice mapper/mapping layer, as well as a potential expression language for handling

post_process()

within the developer flow for taps. Regarding performance and Python/C/OS interop though, a quick dive through Google surfaced https://github.com/pibara/jsonata-wrapper and https://pypi.org/project/pyjsonata/ which both have noted limitations. Also this thread discusses some challenges of Python support on Macs. Wdyt of the Python interop challenges? Have you had any luck as of yet getting something like a POC?

I was thinking to do

tap | nodejs | target

for starters. What would the CLI interface expected for the

nodejs

package published in NPM?

I was thinking to do

tap | nodejs | target

for starters.

@alberto_miorin - Indeed. @ken_payne found a clever solution for evidence-ext which uses the nodejs-bin python wrapper for packaging node into the python ecosystem. From the library description: "With

nodejs-bin

you can call

nodejs

,

npm

and

npx

from both the command line and a Python API." This could be a nice bridge - but of course you could also pipe directly to a nodejs app if you handle the Singer spec parsing and re-forming.

What would the CLI interface expected for the

nodejs

package published in NPM?

The middle layer (we call it a "mapper") would be expected to read Singer input, write Singer output, and accept a

--config=config.json

option in it's CLI. You could implement from scratch, or you could start by forking meltano-map-transform (SDK-based) or pipelinewise-transform-field (not SDK-based) and then add a map function that calls the nodejs transform. In theory, you may simply be able to register a new function called

jsonata()

or similar. The way we do it for

md5()

is to create a helper function here and register it into the

CustomStreamMap

class here. (Subclassing the stream map class may be necessary if you go this route.) The above is just one idea of how to implement. I should add - there are a lot of other ways also. Hope it is helpful!

Thx @aj_steers. I'll check with my team.