hey all, I'm trying to make some tweaks related to retries on a REST stream tap. sometimes the remote server will return an error indicating that the amount of records pulled was too large, this can vary per partition so there's no clear threshold testing the call with a lower

&limit

parameter value seems to solve it, so I'm thinking about having an initial value for the limit on calls and when we hit the error, it would retry but with

&limit={initial limit - 10}

. I'm just not sure what part of the SDK I would have to change to do that.

Hi @joy_yu. RESTStream.backoff_handler should give you access to the

PreparedRequest

object in the

args

key of the details dict. In theory you should be able to mutate this object to reduce the limit by changing the query params.