Question about parallel requests for child streams

Imagine a parent stream that returns a list of contacts and we use their IDs to hit a

/contact_details

endpoint. Supposing I have a page size of 50 in the list of contacts request, I’d like to fire all 50 requests to get the child details at once to speed up the data extraction. Is there a recommended approach for doing parallel requests for child streams on Meltano? I haven’t found any specific mention to that in the docs/community — curious to know how you guys are approaching this.

there's a issue/pr edgar has been thinking about to make this all easier. Today we either batch the requests up and run a clean_up() function at the end of the sync, or during the child_context call we throw the multi threaded call there. Generally I just don't do it until run times are long as pre-optimizing is generally where this question comes from

Got it! It certainly has to be used with caution to avoid hitting API rate limits, but there are scenarios where fetching child streams sequentially one-by-one can cause the entire extraction process to take way longer than it should, especially when there’s more than 1 child stream for the tap.