Hello team 👋 We have a custom tap created using which we want to load the historical records (some 200K) from an API that has Rate Limit of 50 per 10 sec and requires context value from Parent stream to call this child API. This tap handles post process as well where we are appending parent context value with a record to have a unique ID associated with a record. Currently the process is sequential which is impacting the performance especially the full refresh. I can thing of a way to solve it by making async request where we can request in burst of 50 requests per 10 sec instead of making one request and waiting for its response before making second request and doing it 50 times over for each request. Looking for some suggestions or existing implementations. 🙂

Hi @ananya! Is the rate limit global or per endpoint? That’s relevant because in the latter case it could certainly be useful to make as many requests as possible for the parent before, and only start making child requests when the limit is reached. Give #183 a 👍 and/or leave a comment if you’re interested in the feature 😄

@edgar_ramirez_mondragon hey! 👋 do we have rough estimations on when this feature will be released?

@taylor wdyt? 👆🏽 It’s probably not a trivial update, and not guaranteed to improve tap output rate in general.

@ananya this would likely be a post 1.0 addition to the SDK. Given our current priorities I don’t see us adding something like this until next year, but that’s subject to change as we talk to more users in the community. cc @aaronsteers

okay! 👍 looking forward to see this feature added soon 😄 thanks to both of you for chiming in 🙏