Is there a way to run all streams in a tap in para...
# singer-tap-development
m
Is there a way to run all streams in a tap in parallel rather than one stream at a time? The scenario is that I have 3 endpoints to get data from, but each one is an endpoint where I need to • generate report • wait for report to be generated • download and process the report The generate report step takes significant time on some of them, so ideally I could execute the generate report endpoints all at once, then each stream can wait for the report to finish
a
I think the normal way would be a to create 3 definitions in your meltano.yml - each would ‘select’ a different stream. There's no internal parallel execution in meltano, or as part or ‘meltano run’ yet
e
Aaron’s suggestion is the current best way to do it. sdk#183 was logged a while ago, so feel free to leave a 👍 and/or a comment 🙂