https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
j

jeffrey_lewis

06/23/2023, 1:31 PM
I've seen some similar examples by searching this channel but I'm having a hard time getting my head around pagination. I think I may have a really easy use case - my api doc is here by default the API provides 10 records, and then there is a startingAfter parameter with which i can pass the key from the last record of a page into the subsequent call to get the next page. I also have an boolean called "hasmore" that tells me if it's the last page. I can see that I need to have a check if hasmore = true somewhere, then I need to parse the last id in the stream, then i need to include that id in the next call as "startingAfter = id" my url_base method:
Copy code
records_jsonpath = "$[*]"  # Or override `parse_response`.

    # Set this value or override `get_new_paginator`.
    next_page_token_jsonpath = "$.results.id"  # noqa: S105
my get_url_params method:
Copy code
params: dict = {}
        if next_page_token:
            params["startingAfter"] = next_page_token
these are the only real changes i've made in my client.py