https://linen.dev logo
Join Slack
Powered by
Has anyone implemented pagination with an API that...
# singer-tap-development
w
Has anyone implemented pagination with an API that uses x-max-page-size and x-total-count headers? I am using the meltano SDK and am not familiar with how to implement the built in pagination method in client.py
✅ 1
e
hi @william chaplin! are those response headers?
w
@Edgar Ramírez (Arch.dev) Yes, they are response headers.
e
Ok, so I'm trying to understand how you would use those headers to request the name. Can you describe how you would iterate over the pages, e.g. "request the first with 
?page_size=100
then request the next page with 
?page_size=100&page=2
? (or is there a link to public API docs?)
(if they're response headers, your use case is hopefully covered by one of the existing classes)
w
You are correct, in our other integrations we use that pretty much exactly. ?pagesize=500&page=2
Unfortunately the API docs are behind an NDA so I cannot share directly.
👍 1
All of our applications have been c# in the past, we are experimenting with meltano and python right now. No one on the team is an expert in python so were struggling a bit.
e
Ok, so you probably can use BasePageNumberPaginator and get_url_params:
Copy code
class MyPaginator(BasePageNumberPaginator):
    def has_more(self, response):
       return <use the response to determine if the api has any more pages and return True or False>


class MyStream(RESTStream):
    def get_new_paginator(self):
        return MyPaginator(1)

    def get_url_params(self, context, next_page_token):
        return {"pagesize": 500, "page": next_page_token or 1}
👍 1
w
Thank you, I will try this now.
👍 1
@Edgar Ramírez (Arch.dev) Out of curiosity do you happen to have any good resources for learning about the structure of these Cookie cutter projects? I am not sure if its my limited grasp of python but I am having a difficult time understand how the classes and functions are working together here. Right now I have finished all of the getting started content and am currently reading through the Class reference: https://sdk.meltano.com/en/latest/reference.html
@Edgar Ramírez (Arch.dev) Thank you, I ended up implementing the method very similar to how you described and it is working. def has_more(self, response): print(response.headers) if("X-Total-Count" not in response.headers): return False if(int(response.headers['X-Total-Count']) < int(response.headers["X-Max-Page-Size"])): print("Total: " + response.headers['X-Total-Count']) return False return True
🙌 1
10 Views
Open in Slack
PreviousNext
Powered by