https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
a

ananya

09/08/2022, 8:45 AM
Hello team 👋 We have a custom tap created using which we want to load the historical records (some 200K) from an API that has Rate Limit of 50 per 10 sec and requires context value from Parent stream to call this child API. This tap handles post process as well where we are appending parent context value with a record to have a unique ID associated with a record. Currently the process is sequential which is impacting the performance especially the full refresh. I can thing of a way to solve it by making async request where we can request in burst of 50 requests per 10 sec instead of making one request and waiting for its response before making second request and doing it 50 times over for each request. Looking for some suggestions or existing implementations. 🙂