https://linen.dev logo
Join Slack
Powered by
Hi all, I need a bit of help getting pagination wo...
# singer-tap-development
b
Hi all, I need a bit of help getting pagination working for a tap I’m developing for this endpoint using the SDK: https://api.cqc.org.uk/public/v1/changes/providers. The api returns a 
nextPageUri
like this 
changes/provider?page=2&perPage=1000&startTimestamp=2019-03-10T12:30:00Z&endTimestamp=2019-03-24T12:30:00Z&partnerCode=Birdie+Care
and the field is empty for the last page. But setting 
next_page_token_jsonpath = "$.nextPageUri"
gives the following error. “Loop detected in pagination. Pagination token /changes/provider?page=2&perPage=1000&startTimestamp=2019-03-10T123000Z&endTimestamp=2019-03-24T123000Z&partnerCode=Birdie+Care is identical to prior token.” The tap is also outputting the results from the first page of results twice 🤷 . I’m wondering if it’s something configured elsewhere in the tap that is causing this. Any tips on how to debug would be most welcome!
… Also. Setting 
next_page_token_jsonpath = None
works as expected - only the first page of results is downloaded and the tap ends without error.
a
Hi, @benw-at-birdie! Thanks for sharing the details here. The error 
Loop detected in pagination
 is triggered whenever the same next page token is returned twice in a row. In your case, can you confirm if the 'page=2' portion is indeed coming back as the next page token, even on the second page?
If so, meaning, that the way your API shows "end of pagination" is to send the same/self pointer as the page you are already on, then you may need to change the behavior so "loop detection" just signals end of stream.
But before taking that behavior, would be good to check to see if 
page 2
is indeed returning the same token pointing back to itself again as next page. Another possibility - and this is most likely - is that you are getting "page 1" twice in a row, in which case you may need to change how you override "get_url_params()" to make sure it sends the proper request for page two. This probably is going to require some advanced parsing of that token to set the correct url args.
Postman could be helpful in debugging which args are needed. And if you use VS Code, ThunderClient extension provides similar testability natively in VS Code.
b
Thanks AJ 🙏. I’ll take a look this weekend!
Ok worked it out. I wasn’t setting the page parameter in the get_url_params() function - just a typical noob error where i didn’t fully understand what to do 🙂. Now works perfectly! Thanks again.
a
Fantastic! 🙌
Open in Slack
PreviousNext
Powered by