I am developing a tap for an API that will return ...
# singer-tap-development
a
I am developing a tap for an API that will return a HTTP 429 if the rate limit is exceeded. 1. Any suggestions how I might simulate a 429? I can do it with fiddler I suppose--is anything baked into the SDK 2. Is there a recommended practice using the SDK for handling 429 or other error codes?
d
cc @aaronsteers
v
I'm not sure what the goto library is, but there's certainly some kind of mocking framework. https://github.com/getsentry/responses#basics looks like it'd work https://realpython.com/testing-third-party-apis-with-mocks/ as well, Aj will be much better than me 🙂
p
No idea on 1), but for 2), what i did, i don't know if it's the recommended way, but i reimplemented _`_request_with_backoff`_ You can see the code here, including the code to retry to return false in giveup is what makes it retry with that code https://github.com/decentraland/tap-decentraland-api/blob/d878751c609d56713d7a34fb6268cb8fa55061b2/tap_decentraland_api/coingecko_mana_stream.py#L81
p
there are 2 open issues that im aware of around this
a
@pablo_seibelt - As @pnadolny points out, those two issues around rate limits and custom backoff/retry codes would get us closer to handling this in a more streamlined manner. Sounds like you have a path forward though. 👍
p
Yep, just suggesting the current hack that works in case it's useful to Adam, but i agree it'd be nice to have it in the framework, as always thanks for these great tools
a
Thanks for these ideas and the issues.
Any suggestions for handling other error codes, like 500? I'm dealing with a newer API that is not 100% stable
d
a
thanks I will take a look there!
p
On the example i sent, you can handle any errors by specifying the numbers in the giveup
v
https://requests-mock.readthedocs.io/en/latest/ saw while watching a build window scroll with Meltano