https://linen.dev logo
Join Slack
Powered by
```Backing off _request(...) for 4.3s (requests.ex...
# troubleshooting
i
Copy code
Backing off _request(...) for 4.3s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='<http://api.jobdiva.com|api.jobdiva.com>', port=443): Read timed out. (read timeout=300))
I keep getting this error on the 2nd stream of my tap. It runs the first few batches just fine then I get this error. Where can I increase the timeout?
a
Assuming you're using the sdk: https://sdk.meltano.com/en/latest/code_samples.html#custom-backoff might be worth looking at
i
keep-alive: timeout=60
 if my response shows this in the headers, should I set it to 60 instead?
(the response from manually querying the api)
e
You can also set the timeout property of your stream class. The default is 300, so setting it to a lower value wouldn't help.
i
I figured, if this is being caused by some record, how could I just skip the faulty record?
turns out it isn't a bad record - just a huge response
So could I just add it in the tap stream class like:
Copy code
@property
    def timeout(self) -> int:
        return 3600
e
Yup!
i
Sick - let's hope that works hahaha
sidenote - is there a place I can also specify the call limit? it's 120x per minute so I doubt I'll ever hit it but just in case
e
Like a client-side rate limit?
s
I put this logic into tap-rest-api-msdk. It will read the backoff time from the the header or the message response depending on what the tap parameter is set to. I also added a 
backoff_time_extension
as sometimes the reported time to backoff is not long enough, it will add this on top of the recommended backoff time in the API response. https://github.com/s7clarke10/tap-rest-api-msdk/blob/4f87c1adae00446388ebbe418c70b87c231856dc/tap_rest_api_msdk/streams.py#L231-L269
2 Views
Open in Slack
PreviousNext
Powered by