@edgar_ramirez_mondragon you are usually my wizard 🧙 who knows the most about the singer-sdk - Any thoughts?

(ex: exponential)

@edgar_ramirez_mondragon how did you get your bucket to take 3 paramets? It says it only take 2 from reading the docs: RateItem & rate

I thought if I got a 429, I could simply run a wait and try again, but it's not that easy right? Because the pyrate-limiter has no backoff mechanism, and I am replacing the "backoff" decorator with the "limitter" decorator, I am unable to do both at the same time. Therefore, I cannot both use the "limitter" decorator & the "backoff" decorator, and would need to *handle backoff in request.ge*t function call, wherever that is. Quite annoying 😅

Update - thought I had it with this:

def request_decorator(self, func: Callable[..., Any]) -> Callable[..., Any]:
        """Decorate the request method of the stream.

        Args:
            func: The RESTStream._request method.

        Returns:
            Decorated method.
        """
        return limiter.as_decorator()(self._limiter_mapping)(
            super().request_decorator
        )(func)

    def _limiter_mapping(self, *_args: Any, **_kwargs: Any) -> tuple[str, int]:
        return self.tap_name, 1

But what ends up happening is simply that only the

super().request_decorator

is applied

def request_decorator(self, func: Callable[..., Any]) -> Callable[..., Any]:
        """Decorate the request method of the stream.

        Args:
            func: The RESTStream._request method.

        Returns:
            Decorated method.
        """
        return limiter.as_decorator()(self._limiter_mapping)(
            super().request_decorator(func)
            # func
        )

Testing out, but now it should work 😄

Confirmed -> This works 😄

So for posterity, here is the full synopsis: If you wish to apply a logic which combines client-side rate limiting & backoff-error handling, you can do so by using the pyrate library, applying a delay and modifying the request_decorator function in your client. You are essentially adding a decorator on a decorator - this is known as decorator chaining. Here is an example of what I applied:

def request_decorator(self, func: Callable[..., Any]) -> Callable[..., Any]:
        """Decorate the request method of the stream.

        Args:
            func: The RESTStream._request method.

        Returns:
            Decorated method.
        """
        return limiter.as_decorator()(self._limiter_mapping)(
            super().request_decorator(func)
            # func
        )

    def _limiter_mapping(self, *_args: Any, **_kwargs: Any) -> tuple[str, int]:
        return self.tap_name, 1

Note: It's important to note the order here - I want the backoff / error handling to be applied first so that it can handle any errors coming in from the API and only send clean info to the limiter

@edgar_ramirez_mondragon from your experience, how much could this query affect performance? I'm trying to see why, when applying the rate limiting, my speed went down by a factor of 4X. I'm wondering if it's 1. Pour setup 2. Compute heavy