Hi all, I am fairly new to Meltano and wondering, if the following use-case can be done with it: 1. Fetch json files from a filesystem (content: some text) 2. send the individual files to a REST API which applies an NLP model to the content 3. take the answer and write it back to a postgresDB My first guess would be to have a tap for 1. Then chain that into another tap for the REST-API and finally have a targest-postgres. Is something like this possible?

That should be possible with some work, I'd think. Item #2 (calling a REST API) would be a mapper, I think. No such mapper exists as far as I know, so you'd probably have to build that, but I'm not a mapper expert so I could be wrong.

Hey @thomas_briggs using the mapper for that is an interesting idea. I'm not sure that's one of the intended use cases, but hey it might work and can't see why not! @leonard_mandtler keen to follow how you get on.

Hi @thomas_briggs: thanks for the hint on mappers. I just looked it up in the SDK documentation. Under out-of-scope capabilities, there is this part listed:

Mappers do not support external API lookups.
To add external API lookups, you can either (a) land all your data and then joins using a transformation tool like dbt, or (b) create a custom mapper plugin with inline lookup logic.
<https://sdk.meltano.com/en/latest/stream_maps.html>

So I guess, I will look into building my own Mapper plugin with inline lookup logic.

I believe the docs you referenced are for the inline mapping capabilities of the SDK. That is designed to allow limited transformations to be defined as part of a tap or target without additional code, I believe. For what you're describing I think a custom mapper is the right approach. I'm not sure parent-child streams are necessary but I could very well be wrong... I haven't used them myself.

@aaron_phethean, as you asked for an update: here it is. I decided against using Meltano for now. The other parts of our pipeline use own custom code (which is not the best, but hey.) For the first version of this, I will also implement my own worker (which is mostly done already). However, I will look into this again further down the road to see, if the mapper is a legit idea.

Thanks for the update Leonard. Sounds like a great idea to hack together as an example at some point. If / when we do I'll ping you a note. See if it's close to what you implemented in the end. 👍