#3672 feat(taps): Respect rate-limit headers in the default REST backoff Pull request opened by Nazib65 What & why Closes #2012. The default

RESTStream.backoff_wait_generator

returned

backoff.expo(factor=2)

and never looked at the response, so taps hitting

429 Too Many Requests

ignored the server's own

Retry-After

/

X-RateLimit-Reset

headers and waited a guessed exponential interval instead. Per the discussion with @ReubenFrankel and @edgarrmondragon, this implements header-aware backoff within the existing mechanism (the native-retries refactor is left for later given the open

RetriableAPIError

questions), respecting rate-limit headers when present and falling back to exponential backoff when not. Changes •

backoff_wait_generator

is now a generator using backoff's prime-then-

.send(exception)

protocol. It reads the response defensively (

getattr(exception, "response", None)

) so connection/timeout errors that carry no response fall back cleanly instead of raising

AttributeError

. • New overridable

get_wait_time_from_response(response)

hook reads

Retry-After

(precedence; seconds or HTTP-date) then

X-RateLimit-Reset

. • New

_parse_retry_after

helper for the two

Retry-After

value forms. Decisions worth a look •

X-RateLimit-Reset

is treated as seconds-until-reset (delta) per the IETF RateLimit header fields draft referenced in the issue, not a Unix epoch timestamp. Happy to switch to epoch (or support both) if you'd prefer — it's a small change. •

Retry-After

takes precedence over `X-RateLimit-Reset`;

0

-second waits are honoured. • Fully backwards compatible — taps overriding

backoff_wait_generator

are unaffected. Tests Added to `tests/core/rest/test_failure.py`: •

get_wait_time_from_response

parsing: Retry-After seconds + HTTP-date, X-RateLimit-Reset, precedence, and unparsable/missing → exponential fallback. • End-to-end generator drive: header → respected wait;

ConnectionError

(no response) → exponential fallback *without crashing*; header-less

429

→ exponential continues. Full

tests/core/rest/

passes (89);

ruff

,

ruff format

,

ty

, and

mypy

are clean. 🤖 Generated with Claude Code Summary by Sourcery Respect HTTP rate-limit headers when computing REST retry backoff and add tests to verify the new behavior. New Features: • Introduce a header-aware backoff wait generator that prefers server-provided wait times and falls back to exponential backoff. • Add an overridable hook to derive retry wait time from HTTP responses, including support for standard Retry-After and X-RateLimit-Reset headers. Enhancements: • Improve robustness of REST backoff handling for errors without HTTP responses by safely falling back to exponential backoff. Tests: • Add unit tests covering wait-time derivation from rate-limit headers, HTTP-date Retry-After parsing, and the interaction between header-based waits and exponential fallback in the backoff generator. meltano/sdk