2025-02-04T18:44:21.577497Z [info     ] 2025-02-04 13:44:21,576 | INFO     | tap-litmos.lt_UserDetails | Pagination stopped after 0 pages because no records were found in the last response

Where do I need to alter this to continue paginating even if no records were returned for a particular week? I looked into the BasiAPIPaginator and I didn't find where this message is printed. Here's my current paginator:

class LitmosPaginator(BaseAPIPaginator):

    def __init__(self, *args, **kwargs):
        super().__init__(None, *args, **kwargs)

    def has_more(self, response):
        return self.get_next(response) < date.today()

    def get_next(self, response):
        params = dict(parse_qsl(urlparse(response.request.url).query))
        return datetime.strptime(params["to"], OUTPUT_DATE_FORMAT).date() + timedelta(seconds=1)

I'm paginating via a 1 week date range and even if there were no records I still want to move on to the next range. edit: would it be in "advance"? def advance(self, response: requests.Response) -> None: ... ... # Stop if new value None, empty string, 0, etc. if not new_value: self._finished = True else: self._value = new_value

Looks like this is part of _HTTPStream which RESTStream inherits from. You could override all of

RESTStream#request_records

, but you’re likely better off opening an issue and making this behavior more easily override-able without copying most of a large method.

I think for now I'm just going to override the method with:

try:
    first_record = next(records)
except StopIteration:
    self.logger.info(
        "Pagination stopped after %d pages because no records were "
        "found in the last response",
        pages,
    )
    paginator.advance(resp)
    continue

will open an issue too though.