https://linen.dev logo
Join Slack
Powered by
HI Team, i wanted to implement pagination but i am...
# troubleshooting
m
HI Team, i wanted to implement pagination but i am ending up having the error RuntimeError: Loop detected in pagination. Pagination token /data_api/v1/messages?created_since=2022-10-21T16%3A03%3A49.031000&created_to=2022-10-27T14%3A52%3A00&page_size=10000 is identical to prior token.
a
Welcome, @mischal! Check out this recent thread 🧵 - it should be helpful: https://meltano.slack.com/archives/C01PKLU5D1R/p1665773450433149?thread_ts=1665773450.433149&cid=C01PKLU5D1R
m
Hi Aj
in my case its a big different /data_api/v1/messages?created_since=2022-10-21T16%3A03%3A49.031000&created_to=2022-10-27T14%3A52%3A00&page_size=10000
the created_since values gets changed and comes s next_page_uri
but how to override it
i have added the basic functions integrated providing the base url
and $next.page_uri
a
Thanks for the additional details and clarification. Does it look like this is a case of reaching the last page, and the API giving the same "next.page_uri" again after the last page?
If so, and if you can detect the last page on the basis of the API giving a new next page token that's the same as the last, you can force the loop to end by forcing a null (None) return value in get_next_page_token().
m
i did that but that isnt corect becoz what i did was initial path=/messages?updated_since=2022-10-22Ti45200&page_size=100"
this once paginated by using normal pagination in python will give me next_uri_page:- like
next_uri_page:- /data_api/v1/messages?created_since=2022-10-20T16%3A23%3A46.901000&created_to=2022-10-27T14%3A52%3A00&page_size=100
i get the first one but doesnot go the next one after that
ends up with the error same token error
actually i do have more than 1000records in total
does that make sense
r
Let me check my understanding: 1. Initial request to 
/data_api/v1/messages?updated_since=2022-10-22Ti4:52:00&page_size=100
2. Successful response from API with 
next_uri_page
of 
/data_api/v1/messages?created_since=2022-10-20T16%3A23%3A46.901000&created_to=2022-10-27T14%3A52%3A00&page_size=100
3. Next request to 
next_uri_page
value (
/data_api/v1/messages?created_since=2022-10-20T16%3A23%3A46.901000&created_to=2022-10-27T14%3A52%3A00&page_size=100
) 4. Successful response from API with same 
next_uri_page
value as previous response (
/data_api/v1/messages?created_since=2022-10-20T16%3A23%3A46.901000&created_to=2022-10-27T14%3A52%3A00&page_size=100
) 5. Tap throws 
RuntimeError: Loop detected in pagination. Pagination token /data_api/v1/messages?created_since=2022-10-20T16%3A23%3A46.901000&created_to=2022-10-27T14%3A52%3A00&page_size=100 is identical to prior token.
Doesn't sound like an SDK issue. Can you use something like Postman or cURL to manually verify the two requests do not contain the same 
next_uri_page
value in the response body?
I assume in your base tap stream class you have something like
Copy code
next_page_token_jsonpath = "$.next_uri_page"
and
Copy code
def get_next_page_token(
            self, response: requests.Response, previous_token: Optional[Any]
        ) -> Optional[Any]:
            """Return a token for identifying next page or None if no more pages."""
            all_matches = extract_jsonpath(self.next_page_token_jsonpath, response.json())
            return next(iter(all_matches), None)
Open in Slack
PreviousNext
Powered by