https://linen.dev logo
Join Slack
Powered by
Guys, Has anyone performed pagination with the re...
# troubleshooting
c
Guys, Has anyone performed pagination with the rest-api-msdk(https://github.com/Widen/tap-rest-api-msdk) lib? I'm having a lot of problems to be able to paginate a simple API with the PokeAPI (https://pokeapi.co/) that has this pagination model? Can someone help me by telling me how I could configure the parameters of Singer or Meltano to be able to paginate? { "count": 1154, "next": "https://pokeapi.co/api/v2/pokemon?offset=20&limit=20", "previous": null, }
e
Hi @cicero_moura! To make it a bit easier for folks to help you, what have you tried so far and what problems are you having? It seems like you need the 
style1
pagination? https://github.com/Widen/tap-rest-api-msdk#additional-request-styles
j
Hey @edgar_ramirez_mondragon. Me and @cicero_moura work together and we're facing this issue together at this moment. What's going on is that we've tried to use style1 as a pagination method but that isn't working. It's not clear enough how to use it. We're using pokeapi.com in our tests, and we already tried the 
pagination_request_style
in addition to 
ǹext_page_token_path
. We already tried passing $.next as the token_path, and we're trying to figure out how to use pagination_request_style parameter Looking at the source code we've found that there's no use for 
pagination_request_style
parameter inside the methods. Maybe @josh_lloyd can help us with this, as he's the one who developed the connector.
j
pagination is probably the least robust features in this tap because each api seems to have their own way of handling pagination. I wasn’t actually the one that developed that pagination style. you might ask @fred_reimer for more clarification. but I’ll keep looking at this in hopes I can provide some guidance too
there are some unit tests in the tap that you might be able to use to understand better in case you haven’t seen them before https://github.com/Widen/tap-rest-api-msdk/blob/main/tests/test_streams.py
I will admit the documentation could use some clarification as well as the code
the way the code works seems to assume that if you’re using style1 for 
pagination_response_style
that you’re also using style1 for 
pagination_request_style
so if a “next page request” doesn’t use 
offset
and 
limit
as parameters in the request, this style isn’t going to work for you
For the pokeapi, I would assume that a configuration like:
Copy code
pagination_page_size: 20
pagination_response_style: style1
should get you what you want. Is that not the case?
actually looking closer at https://pokeapi.co/docs/v2#resource-listspagination-section makes me wonder if you really just need the default style because the example it provided indicates a 
next
key with a url for the next page in the list. That would indicate that a config of:
Copy code
next_page_token_path: "$.next"
should work
j
@josh_lloyd I was trying some combinations over here and: When I try pagination_page_size with next_page_token_path I ended with the following error:
Here's my configs:
And using these 2 parameters + pagination_response_style, I got just 20 records extracted:
Am I missing something?
j
interesting. I don’t think the 
next_page_token_path
was meant to be used with either 
pagination_page_token
or 
pagination_response_style
can you try it with just 
next_page_token_path
?
j
Sure. The results are the same:
We've already tried these combinations before, I'm just reporting it here right now. This is why we're struggling with these configurations
j
gotcha. Please submit an issue in the repo. I can work on a fix in coming weeks or you’re welcome to submit a PR
j
Alright Josh, we'll see what we can do over here. I really appreciate your attention and your support. Thank you very much. I think we can submit an issue in the next days
2 Views
Open in Slack
PreviousNext
Powered by