Hi all, I am trying to use tap-rest-api-msdk to fetch data from a Rest API but I am not able to get the pagination correct. I think it's something incorrect in configuration. Can someone help me? Here is the yml config -

- name: tap-rest-api-msdk--xyz
    inherit_from: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://api.xyz.io/v2>
      headers:
        Authorization: API_KEY
        accept: application/json
      pagination_request_style: offset_paginator
      pagination_response_style: style1
      pagination_page_size: 100
      pagination_limit_per_page_param: per_page
      pagination_total_limit_param: total_count
      next_page_token_path: $.next_page_link
      backoff_time_extension: 20
      streams:
        - name: calls
          path: /calls
          primary_keys:
            - id
          records_path: $.data[*]
          num_inference_records: 50
    select:
    - '*.*'

Have you tried changing the paginator_request_style to

jsonpath_paginator

? You may need to also experiment with different combinations of the response style as well e.g.

hateoas_body

.