Hello All, I am building a tap for a rest api and trying to implement the pagination but it seems like not working. The next_page_info is in the json response in the string form .

"metadata": {

"next_page_info": "xxxxxxxxxxxxxxxx"

}

Error i am getting:

simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Code i am using:

records_jsonpath = "$[*]"  # Or override `parse_response`.
next_page_token_jsonpath = "$.next_page_info"  # Or override `get_next_page_token`.


    def get_next_page_token(
        self, response: requests.Response, previous_token: Optional[Any]
    ) -> Optional[Any]:
        """Return a token for identifying next page or None if no more pages."""
        # TODO: If pagination is required, return a token which can be used to get the
        #       next page. If this is the final page, return "None" to end the
        #       pagination loop.
        if self.next_page_token_jsonpath:
            all_matches = extract_jsonpath(
                self.next_page_token_jsonpath, response.json()
            )
            print("all match ", all_matches)
            first_match = next(iter(all_matches), None)
            next_page_token = first_match
            print("next_page_token....................", next_page_token)
        else:
            next_page_token = response.headers.get("page_info", None)

        return next_page_token



    def get_url_params(
        self, context: Optional[dict], next_page_token: Optional[Any]
    ) -> Dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization."""
        params: dict = {
        }
        if next_page_token:
            params["page_info"] = next_page_token
            params["per_page"] = 100
        if self.replication_key:
            params["sort"] = "asc"
            params["order_by"] = self.replication_key
        print("params ", params)
      
        return params

Shouldn't your next_page_token_jsonpath be $.metadata.next_page_info

Did this change and it's able to get the

next_page_info

token but still getting the error

ERROR Loading failed (1): simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

meltano | DEBUG ELT could not be completed: Loader failed I think something wrong in my json schema.

This error looks like it is not related to jsonschema validation actually The API is not returning valid json and failing during deserialization, thats what the error is saying in this case.

Okay i am checking.

If this was your next page path (

$.metadata.next_page_info

), it is impossible for the existing record path (

$[*]

) to be correct Use this if you need to validate or figure out how jsonpath's evaluate, keep us posted on success!

Yes it was working, i was able to get the next_page_info but after getting all next_page_info , it's throwing

ERROR Unable to parse JSON: next_page_token.................... MTA0Njc4MDkyMiwxNjU5NDU2NDI2

( This is the first token) Below is the screenshot of response.

@monika_rajput that error seems to be coming from the target. There’s something in stdout of the tap that’s not valid json. Perhaps you’re `print`ing some debug message in the tap? Something like

meltano invoke <your tap> > singer.jsonl

would confirm by looking at which lines are emitted by the tap.

yes using --log-level

meltano --log-level=debug elt tap-yotpo redshift-pipelinewise

Did you validate your record jsonpath (

$[*]

) against a sample response from the API using this tool I linked before? Need to rule this out just in case

Yes i validate the response.Using

$.customers[*]

So did you update your

records_jsonpath

in your tap to

$.customers[*]

❓

Yes i updated and it is resolved, now it's working fine . Thank you so much😊