Hi, I am looking to build a workflow of metalno/singer tap. I am thinking of two main use case: i) being able to run two taos in parallel, ii) running them sequentially, having one feeding the next. Could you advise if a python framework or methodology that could help? Thank you Fred

Hi @Frederic! The SDK docs might be a useful resource if you're interested in tap development 🙂

being able to run two taos in parallel

Do you mean with a single Meltano command?

running them sequentially, having one feeding the next

Can you say more about this use case? It sounds interesting, I'll try to dig up old discussions about making some plugins "ambidextrous", i.e. making them capable of reading and writing data from a particular system.

Thank you Edgar, For the second use case I found something interesting here: https://github.com/singer-io/singer-python/issues/84. This is an old article from 2018, perhaps there has been an improvement. The use case example would like this. • tap 1: gets a list of items from a web api • Tap2: uses the output list from tap 1, as input into the tap2 for further processing.

Oh yeah, that's totally been discussed before. It's a major architectural change to Meltano itself, and external orchestrators might be capable of doing something like this. Do log an issue in the Meltano repo if you have suggestions or other ideas 🙏

Thank you for the prompt reply

Tons of us have already done this in different ways, so it works. Specifically for your tap1/tap2 combo I'd use a setting (config) to input data for one of the streams for further processing. Really depends on the use case. Lots of folks use the orchestrator to fan out and settings/config to control how to run them. For some jobs, it makes sense to parallelize in the tap itself.

Thank you for your response