Is there some more info on how to implement paging? I am new to singer tap development and try to implement the following call right now: https://docs.bexio.com/#tag/Timesheets with a limit and offset on the call, but I have no idea how to actually implement this. Basically I already started with extending the get_url_params with the right params:

def get_url_params(
        self, context: Optional[dict], next_page_token: Optional[Any]
    ) -> Dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization."""
        params: dict = {}
        if next_page_token:
            params["offset"] = next_page_token
            params["limit"] = 500
        if self.replication_key:
            params["sort"] = "asc"
            params["order_by"] = self.replication_key
        return params

Not sure what to do with `next_page_token_jsonpath = "$.next_page" # Or override `get_next_page_token`` or

def get_next_page_token(
        self, response: requests.Response, previous_token: Optional[Any]
    ) -> Optional[Any]:
        """Return a token for identifying next page or None if no more pages."""
        # TODO: If pagination is required, return a token which can be used to get the
        #       next page. If this is the final page, return "None" to end the
        #       pagination loop.
        if self.next_page_token_jsonpath:
            all_matches = extract_jsonpath(
                self.next_page_token_jsonpath, response.json()
            )
            first_match = next(iter(all_matches), None)
            next_page_token = first_match
        else:
            next_page_token = response.headers.get("X-Next-Page", None)

        return next_page_token

I'd recommend starting from the api response / docs, and working from there. What is the next page token the api requires you to use? Where does that come from in the json response? Once you answer those then you will know what you need the sdk to do, then it's just implementation details

Thanks for always helping me @visch 😄 That's the thing, I went through the api documentation and I can't find a token indicating if there is a next page or not in any of the responses. There is just the offset and limit params in the url to call.

No problem @nino_mueller they are good questions! Well you have to figure out the source api first, there's no way the sdk is going to figure it out first (in today's world 😄 ) , offset and limit params are somewhat of a common pattern. https://github.com/AutoIDM/tap-zohosprints/blob/main/tap_zohosprints/streams.py#L270 kind of follows this (Note that this tap has not been cleaned up so there's some hanging code, I'd just focus on this line here, next page token is fortunately easy here in this api https://github.com/AutoIDM/tap-zohosprints/blob/main/tap_zohosprints/client.py#L47 .

So if I check out Zoho Sprints API it looks like there is a next param that can be used. Bexio API offers no info on the next page.. Would it be possible to change the `next_page_token_jsonpath = "[]" # Or override `get_next_page_token`` so that it goes to the next page until we receive an empty array and then we stop?

yes, you just have to define what "next page" is and tell it when to stop 😄

Yes, it's pretty easy and fast to create a singer tap with meltano sdk 🙂 I already implemented it now by checking the response header for the content-length of 2 (for the empty array). Not the most elegant solution, but I didn't see another way because the API is not offering me a way to know if i am on the last page without asking 😄 https://github.com/substringgmbh/tap-bexio/blob/2724c2d5076312740d5b64e4261d45a2d7ed9a50/tap_bexio/client.py#L42

slightly dangerous, but 🤷 if it works. I'd check an empty result set instead as there's probably more reasons than 1 for a content-length of 2, but it doesn't matter so much

I am still using lots of magic numbers and need to go through it haha 😄 500 is the default page size, that's why I used it.