I've seen some similar examples by searching this channel but I'm having a hard time getting my head around pagination. I think I may have a really easy use case - my api doc is here by default the API provides 10 records, and then there is a startingAfter parameter with which i can pass the key from the last record of a page into the subsequent call to get the next page. I also have an boolean called "hasmore" that tells me if it's the last page. I can see that I need to have a check if hasmore = true somewhere, then I need to parse the last id in the stream, then i need to include that id in the next call as "startingAfter = id" my url_base method:

records_jsonpath = "$[*]"  # Or override `parse_response`.

    # Set this value or override `get_new_paginator`.
    next_page_token_jsonpath = "$.results.id"  # noqa: S105

my get_url_params method:

params: dict = {}
        if next_page_token:
            params["startingAfter"] = next_page_token

these are the only real changes i've made in my client.py

I would recommend that you use a custom paginator class to do this:

pagination.py

class ruddrPaginator(BaseAPIPaginator):
    def get_next(self, response):
        body: dict = response.json()
        hasMore = body.get("hasMore")

        # if no more results, stop pagination
        if not hasMore:
            return

        # leverage JSONPathPaginator to lookup the id of the last result
        return JSONPathPaginator("$.results[-1:].id").get_next(response)

client.py

def get_new_paginator(self):
        return ruddrPaginator()

This is similar to what we do in `tap-auth0`: https://github.com/Matatika/tap-auth0/blob/5f9e34e7d705fe9a72bec6be28a9b4cf53bc4fc4/tap_auth0/pagination.py https://github.com/Matatika/tap-auth0/blob/5f9e34e7d705fe9a72bec6be28a9b4cf53bc4fc4/tap_auth0/client.py#L38-L39

thank you! this makes sense. i'll give it a try

Anecdotally, in your

client.py

I think you want to set your

records_jsonpath

property to

$.results

to just return the objects from

results

.

totally i've noticed that- i've got my search paths set redundantly in my streams.py to be $.results[*], in streams.py they should be * and in clients.py it should be as you've said there

Ah, gotcha. That works too!

ok so just trial and error here... this runs but it doesn't do what it's supposed to: • pagination • client i think what i'm missing is how i set next_page_token properly...

The JSONPath expression here should be

$.results[-1:].id

, if you want the last result

id

. Currently, the expression probably isn't matching anything in the response body, so

None

is returned.

dude thank you so much for the hint! that and a little testing in the JSON path tester page got it flowing. now i'm back to catching some new schema errors

No problem! Glad you got it working. 😎