Hello everyone 🙂 I'd like to ask about any parallelization options when it comes to separate streams within a single tap. I have a custom tap developed with help of Singer SDK. The tap hits different rest api endpoints per each defined stream and syncs with db sinks (transferwise variant of target-postgres or target-snowflake) within an EL run. Are the different stream syncs handled asynchronously by meltano/singerSDK by any chance? If not, is it possible to support this by modifying the logic of the custom Tap? I am not sure what are the limitations here (or rather, how much support for parallel stream syncs already is in place). I tried consulting documentation and searching in different channels here in meltano slack channel, but with no success so far 😕 any help would be appreciated.

Hi @tomas_jevocin 👋🏼 There's https://github.com/meltano/sdk/issues/183 but no progress has been made in that direction. Making the framework async is a significant change that involves lots of refactoring with tons of unknowns, but I'm open to ideas. In the meantime, one way to create parallel syncs of different streams is to use Meltano's plugin inheritance and selecting different streams in each inherited plugin.

Hi again @edgar_ramirez_mondragon 🙂 thanks for the response. I have some small follow up questions: • So do I understand correctly that if I run multiple loader plugins (in this case inherited ones), they will be run asynchronously? • Would just simply calling the

meltano el ...

be enough to enable parallel execution of multiple loader plugins, or do I have use some other command?

You'd instead run multiple extractor plugins, but otherwise the answer is "yes"

I see, thank you!