Hi everyone, new here 😄 I'm currently using

tap-rest-api-msdk

(variant:

widen

) with Meltano

v3.7.8

, and I’ve encountered an issue related to pagination configuration. Here's the relevant part of my config:

plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://api.yuman.io>
      auth_method: bearer_token
      bearer_token: ${YUMAN_BEARER_TOKEN}
      request_timeout: 300
      request_records_limit: 1000
      backoff_type: header
      backoff_time_extension: 15
      backoff_param: Retry-After
      pagination_request_style: page_number_paginator
      pagination_response_style: default
      pagination_page_size: 50
      pagination_initial_offset: 1
      pagination_limit_per_page_param: per_page
      pagination_page_number_field: page
      next_page_token_path: $.page
      streams:
        - name: yuman_clients
          path: /v1/clients
          records_path: $.items[*]
          primary_keys: [id]
          schema: extract/schemas/yuman_clients.json
          replication_key: updated_at
          params:
            per_page: 50
            embed: fields

        - name: yuman_material_categories
          path: /v1/materials/categories
          records_path: $.items[*]
          primary_keys: [id]
          replication_key: updated_at
          paginator: single_page_paginator

The issue: The pagination config (e.g.,

pagination_request_style

,

pagination_page_size

, etc.) seems to only apply at the top-level

config

. When I try to set a specific pagination strategy at the stream level (like using

paginator: single_page_paginator

for one stream), it appears to be ignored or not working as expected. I have two endpoints with different pagination styles, so I need to customize pagination behavior per stream. I was under the impression this was supported before — has something changed recently in how pagination configs are handled per stream? Is this expected behavior, or should stream-level pagination overrides still work? Thanks in advance for any help or clarification!

Hi @Ceb 👋 Looking at their readme, I don't think per-stream pagination configuration was ever supported: https://github.com/Widen/tap-rest-api-msdk/?tab=readme-ov-file#stream-level-config-options. One option is to use inheritance to configure multiple instances of the tap for streams with different pagination logic.

Hi! Thank you for your answer. I guess I’ll go ahead and create another instance of the tap specifically for that endpoint. Thanks again, and have a great day! 😄