Hi again… I’m having a little trouble with pagination on my REST tap. The API returns a json field with a complete link to the next page (not page number, just random token added on to URL). I tried to change the

get_next_page_token

function to return just the

nextToken=XYZ987

part of the url, but I get a “loop detected … identical token” error. Any ideas? The SDK documents a number of different paginators and one seems to address this nextURL pagination method, but I don’t know how to incorporate them in the tap.

That error means that the same token was detected on multiple passes - perhaps can you add logging to print the token value?

Good, at least I understood the message correctly. Do I understand correctly what the function is supposed to return? e.g. for the rickandmorty api example, the function should extract

page=2

from

<https://rickandmortyapi.com/people?page=2>

? I have been debugging the API responses a little closer, and there does indeed seem to be some strange behavior coming from there - so this probably isn’t on the meltano side.

Yes, that's the correct understanding 👍 In that example, it's entirely your choice how to store the token. But

2

or

{page: 2}

both should work. In some scenarios, developers end up with very complex tokens, which they use for managing complex or nested paging loops. This one probably would be fine with a simple token, whatever seems appropriate.

The SDK documents a number of different paginators and one seems to address this nextURL pagination method, but I don’t know how to incorporate them in the tap.

I’ve started a PR to document how to do that: https://github.com/meltano/sdk/pull/1074

Thank you so much @edgar_ramirez_mondragon, that was exactly the example I needed to get my pagination up and running. My tap is still in it’s early stages, but if nothing else just to show the pagination implemented: https://github.com/radbrt/tap-sharepointsites