Is there an example anywhere for handling backoff ...
# singer-tap-developmentr
Reuben (Matatika)
10/06/2023, 4:17 PMIs there an example anywhere for handling backoff with an
X-RateLimit-Resete
edgar_ramirez_mondragon
10/06/2023, 4:24 PMr
Reuben (Matatika)
10/06/2023, 10:02 PMI tried implementing this with
backoff.runtimee
edgar_ramirez_mondragon
10/06/2023, 10:09 PMProbably. The stream class has a
backoff_jitterr
Reuben (Matatika)
10/06/2023, 10:45 PMI thought this might fix it
Copy code
def backoff_jitter(self, value):
self.logger.info(f"Jitter: {value}")
return value Copy code
2023-10-06 23:40:53,555 INFO METRIC: {"metric_type": "timer", "metric": "http_request_duration", "value": 0.068145, "tags": {"stream": "stream_auth0_logs", "endpoint": "/logs", "http_status_code": 429, "status": "failed"}}
2023-10-06 23:40:53,555 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1696632114
2023-10-06 23:40:53,555 Reset: 1696632114.0 Now: 1696632053.555651 Wait: 60.44434905052185
2023-10-06 23:40:53,555 Backing off _request(...) for 50.1s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /api/v2/logs)
2023-10-06 23:40:53,555 Backing off 50.1 seconds after 1 tries calling function <bound method RESTStream._request of <tap_auth0.streams.LogsStream object at 0x7f3ddc34ec10>> with args (<PreparedRequest [GET]>, None) and kwargs {}
2023-10-06 23:41:43,760 INFO METRIC: {"metric_type": "timer", "metric": "http_request_duration", "value": 0.027469, "tags": {"stream": "stream_auth0_logs", "endpoint": "/logs", "http_status_code": 429, "status": "failed"}}
2023-10-06 23:41:43,761 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1696632114
2023-10-06 23:41:43,761 Reset: 1696632114.0 Now: 1696632103.761188 Wait: 10.23881196975708
2023-10-06 23:41:43,761 Backing off _request(...) for 2.5s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /api/v2/logs)
2023-10-06 23:41:43,761 Backing off 2.5 seconds after 2 tries calling function <bound method RESTStream._request of <tap_auth0.streams.LogsStream object at 0x7f3ddc34ec10>> with args (<PreparedRequest [GET]>, None) and kwargs {}
2023-10-06 23:41:46,299 INFO METRIC: {"metric_type": "timer", "metric": "http_request_duration", "value": 0.047897, "tags": {"stream": "stream_auth0_logs", "endpoint": "/logs", "http_status_code": 429, "status": "failed"}}
2023-10-06 23:41:46,299 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1696632114
2023-10-06 23:41:46,299 Reset: 1696632114.0 Now: 1696632106.299446 Wait: 7.700553894042969
2023-10-06 23:41:46,299 Backing off _request(...) for 2.3s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /api/v2/logs)
2023-10-06 23:41:46,299 Backing off 2.3 seconds after 3 tries calling function <bound method RESTStream._request of <tap_auth0.streams.LogsStream object at 0x7f3ddc34ec10>> with args (<PreparedRequest [GET]>, None) and kwargs {}
2023-10-06 23:41:48,656 INFO METRIC: {"metric_type": "timer", "metric": "http_request_duration", "value": 0.053343, "tags": {"stream": "stream_auth0_logs", "endpoint": "/logs", "http_status_code": 429, "status": "failed"}}
2023-10-06 23:41:48,656 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1696632114
2023-10-06 23:41:48,656 Reset: 1696632114.0 Now: 1696632108.656409 Wait: 5.343590974807739
2023-10-06 23:41:48,656 Backing off _request(...) for 1.4s (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /api/v2/logs)
2023-10-06 23:41:48,656 Backing off 1.4 seconds after 4 tries calling function <bound method RESTStream._request of <tap_auth0.streams.LogsStream object at 0x7f3ddc34ec10>> with args (<PreparedRequest [GET]>, None) and kwargs {}
2023-10-06 23:41:50,040 INFO METRIC: {"metric_type": "timer", "metric": "http_request_duration", "value": 0.027334, "tags": {"stream": "stream_auth0_logs", "endpoint": "/logs", "http_status_code": 429, "status": "failed"}}
2023-10-06 23:41:50,040 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1696632114
2023-10-06 23:41:50,040 Giving up _request(...) after 5 tries (singer_sdk.exceptions.RetriableAPIError: 429 Client Error: Too Many Requests for path: /api/v2/logs)backoff_jittere
edgar_ramirez_mondragon
10/09/2023, 2:43 PMIt should definitely be called by https://github.com/meltano/sdk/blob/bab2aa9a7ffcdc4887d754e66145849cdfc8f561/singer_sdk/streams/rest.py#L249 and https://github.com/litl/backoff#jitter 🤔
edgar_ramirez_mondragon
10/09/2023, 2:47 PMOh I think you figured it out 😁
r
Reuben (Matatika)
10/09/2023, 2:48 PMYeah that got things working, but ideally I'd like to fix it using
X-RateLimit-ResetReuben (Matatika)
10/09/2023, 2:51 PMI don't see that log message I put in
backoff_jitterReuben (Matatika)
10/09/2023, 2:52 PMEither way, the issue we were having is fixed - this was more a question out of curiosity. 🙂
Reuben (Matatika)
10/10/2023, 5:34 PM@edgar_ramirez_mondragon Would you accept a PR to support
RateLimite
edgar_ramirez_mondragon
10/10/2023, 5:36 PM@Reuben (Matatika) Sure thing! Do you mind creating an issue? Otherwise I can do it later today.
r
Reuben (Matatika)
10/10/2023, 5:36 PMSure, thanks. 🙂