https://linen.dev logo
Join Slack
Powered by
Hi Hi, Just a quick question about pagination. How...
# troubleshooting
h
Hi Hi, Just a quick question about pagination. How would I go about doing the pagination if this is my paging body? 🤔 Any help would be appreciated 🙏
Copy code
:{
    "customers": [
        {
            "customerId": "123"
        },
        {
            "customerId": "234"
        }
    ],
    "paging": {
        "totalCount": 15,
        "totalPages": 4,
        "currentPage": 1
    }
}
✅ 1
v
depends on the request exactly but I'd guess something like 
/api?page=2
/api?page=3
/api?page=4
currentpage == totalPage , done
➕ 1
e
+1 on Derek's comment. It seems pagination is handled completely on the request side, so you probably have to include a 
page
key or similar in the return value of 
get_url_params
.
h
Hey hey sorry for the bother and a bit late on the subject, but I'm still stuck here. Here is my code:
Copy code
class LoyaltyPaginator(BasePageNumberPaginator):
    def get_next(self, response):
        data = response.json()
        totp = data['totalPages']
        curp = data['currentPage']

        modpg = curp%(totp + 1)
        page = modpg + 1 if modpg < totp else None

        return page
I am following https://sdk.meltano.com/en/v0.25.0/guides/pagination-classes.html, but I am getting an error: 
TypeError: BaseAPIPaginator.__init__() missing 1 required positional argument: 'start_value'
any insight would be helpful thanks 🙏
r
I assume you are instantiating 
LoyaltyPaginator
somewhere - you will need to pass the starting page number 
client.py
Copy code
def get_new_paginator(self):
        return LoyaltyPaginator(0)
or implement 
__init__
for 
LoyaltyPaginator
Copy code
class LoyaltyPaginator(BasePageNumberPaginator):

    def __init__(self):
        return super().__init__(0)

    ...
client.py
Copy code
def get_new_paginator(self):
        return LoyaltyPaginator()
🙏 1
h
aah thank you so much 🥳 however I seem to directly run into another error which I don't understand where it could come from. It doesn't seem to recognise the key 'totalPages' 
KeyError: 'totalPages'
This is how the paging body looks like:
Copy code
:{
    "customers": [
        {
            "customerId": "123"
        },
        {
            "customerId": "234"
        }
    ],
    "paging": {
        "totalCount": 15,
        "totalPages": 4,
        "currentPage": 1
    }
}
Any idea on what I might be doing wrong here? 🤔
r
Copy code
class LoyaltyPaginator(BasePageNumberPaginator):
    def get_next(self, response):
        data = response.json()["paging"]  # <---
        totp = data['totalPages']
        curp = data['currentPage']

        modpg = curp%(totp + 1)
        page = modpg + 1 if modpg < totp else None

        return page
👍 2
💯 1
❤️ 1
🙏 1
h
aaah forgot ["paging"], thank you. However I'm still stuck with the error: 
RuntimeError: Loop detected in pagination. Pagination token 2 is identical to prior token.
,but when I test the paging algorithm it runs fine 🤔 thank you for all the help and sorry for all the questions 🙏
r
Copy code
class LoyaltyPaginator(BasePageNumberPaginator):
    def get_next(self, response):
        data = response.json()["paging"]
        totp = data['totalPages']
        curp = data['currentPage']

        page = curp + 1 if curp < totp else None

        return page
I think 
modpg = curp%(totp + 1)
was over-complicating things.
h
unfortunately still getting the same 
RuntimeError: Loop detected in pagination. Pagination token 2 is identical to prior token.
I've modified the paginator class to look like this:
Copy code
class LoyaltyPaginator(BasePageNumberPaginator):
    
    def __init__(self):
        return super().__init__(0)
    
    def get_next(self, response):
        data = response.json()["paging"]
        totp = data['totalPages']
        curp = data['currentPage']

        page = curp + 1 if curp < totp else None

        return page
e
Should the initial pagination value be 1?
h
I've tried setting the starting value to 1 as well, however if the starting value is 0 it also returns the first page of records. Should the following also be added on somewhere into the code:
Copy code
@cached_property
    def authenticator(self):
        return LoyaltyPaginator()
or is it fine otherwise as long as I added:
Copy code
def __init__(self):
        return super().__init__(0)
Into the paginator class like above? This is me returning the paginator in get_new_paginator:
Copy code
def get_new_paginator(self) -> LoyaltyPaginator:
        return LoyaltyPaginator()
and def get_url_params is pretty much the same. I haven't made any changes. 🤔
e
what logs do you see before the error?
r
Sorry, I did a bad copy-paste before. Put back whatever you had for the 
authenticator
property implementation.
👍 1
Are you doing something like
Copy code
params["page"] = next_page_token
in 
get_url_params
?
h
this is get_url_params:
Copy code
def get_url_params(
        self,
        context: dict | None,  # noqa: ARG002
        next_page_token: Any | None,  # noqa: ANN401
    ) -> dict[str, Any]:
        params: dict = {}


        if next_page_token:
            params["page"] = next_page_token

        if self.replication_key:
            params["sort"] = "asc"
            params["order_by"] = self.replication_key
        
        starting_date = self.get_starting_timestamp(context)
        if starting_date:
            params["after"] = starting_date.isoformat()
        
        if next_page_token is not None:
            params["page"] = next_page_token
        
        <http://self.logger.info|self.logger.info>("QUERY PARAMS: %s", params)
        return params
        yield from extract_jsonpath(self.records_jsonpath, input=response.json())
I haven't really made any changes in get_url_params. I didn't make any changes to the authenticator property. I only added the innit def to the paginator 👍 No errors or warnings leading up to the runtime error only errors I get is : 
raise RuntimeError(msg)
RuntimeError: Loop detected in pagination. Pagination token 2 is identical to prior token.
r
Are you sure 
page
is the correct param? I imagine this is what might be happening: 1. Request to 
/api
2. Response containing 
paging.currentPage
value of 
1
3. 
next_page_token
resolved as 
2
4. Request to 
/api?page=2
5. API doesn't respect 
page
param, so it is ignored - effectively the same request to 
/api
as in step 1 6. Response containing 
paging.currentPage
value of 
1
7. 
next_page_token
resolved as 
2
8. 
RuntimeError: Loop detected in pagination. Pagination token 2 is identical to prior token.
h
Hi, hi yes still stuck on this issue. ?page=2 does work in the Api as I use postman and going through the pages using page param does work. So I don't believe it is the issue.
Sorry for delayed replies on the thread. I am working on and off when I get the chance to between studies. 🙏
👍 1
However I did recheck all the logs and notice An unhandled error occurred while syncing (stream) after QUERY PARAMS: {'page': 2}.🤔
r
Try this in your stream class and see what is logged:
Copy code
def validate_response(self, response):
        <http://self.logger.info|self.logger.info>("Paging: %s", response.json()["paging"])
        super().validate_response(response)
Failing that, you should try the same but directly in 
LoyaltyPaginator.get_next
- since there is no 
logger
available, you will have to configure one yourself, or you can do what I do sometimes:
Copy code
def get_next(self, response):
        data = response.json()["paging"]

        import sys
        print(data, file=sys.stderr)

        totp = data['totalPages']
        curp = data['currentPage']

        page = curp + 1 if curp < totp else None

        return page
Otherwise, look into a proper debug setup with breakpoints in your IDE: https://sdk.meltano.com/en/latest/dev_guide.html#ide-tips For example: https://github.com/Matatika/tap-spotify/blob/a6da36f062b32d0f6847cd3eb01b300e35a19e59/.vscode/launch.json
🙏 1
h
Hi a bit of a late reply, but just wanted to say thank you🙏. The error ended up being caused by the last line of code in get_url_params. 
yield from extract_jsonpath(self.records_jsonpath, input=response.json()
r
Oh yeah, that was definitely out of place 😅 No problem!
3 Views
Open in Slack
PreviousNext
Powered by