I'm having a heck of a time trying to get pagination working on tap-rest-api-msdk. The API I'm calling responds with a response header 'Link' with content:

<<https://api-na.myconnectwise.net/v4_6_release/apis/3.0/company/companies/?pageSize=25&page=2>>; rel="next", <<https://api-na.myconnectwise.net/v4_6_release/apis/3.0/company/companies/?pageSize=25&page=64>>; rel="last"

and has nothing but records in the response body - there is no JSON 'pagination' content in the body. I'm not sure if I can reference this with the next_page_token_path construct, as that seems to be associate with JSON response body content? I've been unable to get any combination of pagination_request_style's to work ... typically resulting in either an error of 'list' object has no attribute 'get' when the request style is searching for some sort of JSON object, or just returning the first page by default. Current config is:

version: 1
default_environment: dev
project_id: 41a9e06b-03e8-48d4-a798-78797e882fe7
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      username: [redacted]
      api_url: <https://api-na.myconnectwise.net/v2022_2/apis/3.0>
      auth_method: basic
      headers:
        clientid: 824f139b-27ea-4b1f-abc6-2737d76761cb
      pagination_request_style: offset_paginator
      pagination_response_style: offset
      pagination_limit_per_page_param: pageSize
      streams:
      - name: companies
        path: /company/companies/
      flattening_enabled: true
      flattening_max_depth: 1
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Any advice/assistance anyone could offer would be greatly appreciated. Apologies for lack of errors or more results in this post...

Instead of using

tap-rest-api-msdk

I would go with writing a tap myself tbh as this stuff tends to be complicated. Not saying you couldn't eventually make this work but it's a lot of code in configuration for my taste I don't have much experience with

tap-rest-api-msdk

so I can't help much with the tap side, but the https://sdk.meltano.com/en/latest/ is handy if you want to write a tap yourself 🤷

Yeah, and HATEOAS is a well-supported pattern: https://sdk.meltano.com/en/latest/guides/pagination-classes.html#example-hateoas-pagination-a-k-a-next-links From that example you'd only need to change how the next URL is retrieved, i.e. from the headers instead of the response body: https://requests.readthedocs.io/en/latest/user/advanced#link-headers

Thanks for the wayfinding, guys. I'm currently reviewing the docs and youtube video about building your own tap.