https://linen.dev logo
Join Slack
Powered by
Hi everyone I’m facing an issue with the tap-rest-...
# troubleshooting
k
Hi everyone I’m facing an issue with the tap-rest-api-msdk extractor in Meltano and need some help debugging. My meltano.yml config (relevant part):
Copy code
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://app.leaflink.com/api/v2>
      headers:
        Authorization: App XXX
        Accept: application/json
      pagination_request_style: jsonpath_paginator
      stop_on_empty_next_url: true
      next_page_token_path: $.next

      backoff_type: header
      backoff_param: Retry-After
      backoff_time_extension: 7
      streams:
      - name: orders-received
        path: /orders-received
        records_path: $.results[*]
        schema: .meltano/extractors/schemas/schema.json
        params:
          limit: 100
        
    select:
    - '*.*'
Problem The API I’m calling has ~19,000 records. But when Meltano runs the extractor, it looks like it is sending only ~21 requests and then stops. I verified this manually: • With 
limit=50
→ Meltano gives ~1050 records (≈21 × 50) • With 
limit=100
→ Meltano gives ~2100 records (≈21 × 100) After those 21 requests, Meltano fails with:
Copy code
singer_sdk.exceptions.RetriableAPIError: 502 Server Error: Bad Gateway
1. Interestingly, when I call the same API via Python 
requests
, it works fine and I can fetch all pages. What could be the possible reason and how to resolve it? Any ideas or guidance would be really appreciated
a
Could this be some sort of rate limiting if you always get the error? That shouldn't result in 502s though. When you try the call via python requests, how do you handle pagination? Maybe try removing the
Copy code
backoff_type: header
      backoff_param: Retry-After
      backoff_time_extension: 7
section and see if it is handled better. Somewhere 
RetriableAPIError
is not getting caught correcting.
k
Python script (below) which is successfully executed:
Copy code
import requests
import json
 
API_URL = "<https://app.leaflink.com/api/v2/orders-received>"
 
HEADERS = {
    "Authorization": "App ********",
    "Accept": "application/json"
}
 
def fetch_orders_received():
    all_records = []
    next_url = API_URL  # first request
 
    while next_url:
        print(f"\nFetching: {next_url}")
 
        response = requests.get(next_url, headers=HEADERS)
        response.raise_for_status()
 
        data = response.json()
 
        if "results" not in data:
            print("ERROR: 'results' not found in API response")
            break
 
        records = data["results"]
        print(f"Fetched {len(records)} records")
 
        all_records.extend(records)
 
        next_url = data.get("next")
 
        if not next_url:
            print("Reached last page.")
            break
 
    return all_records
 
 
if __name__ == "__main__":
    records = fetch_orders_received()
 
    print(f"\nTotal records fetched: {len(records)}")
 
    with open("orders_received.jsonl", "w", encoding="utf-8") as f:
        for rec in records:
            f.write(json.dumps(rec) + "\n")
 
    print("Saved to orders_received.jsonl")
Also, I tried your suggestion by removing this block of code: 
backoff_type: header backoff_param: Retry-After backoff_time_extension: 7
but facing same error. Any suggestions why is it happening?
a
What meltano command are you running?
k
Copy code
meltano run tap-rest-api-msdk target-jsonl
@Andy Carter Any help?
a
I don't use this tap myself, but sounds like odd behaviour. Could you post the full stack trace rather than just the final error message? Are you definitely getting 21 pages of different data, rather than the same page over and over? Trying to determine if the pagination is working correctly.
➕ 1
k
@Andy Carter I just observed that I am getting same data page over and over. So the config for the pagination is not working. Can you please help me what should be the config to make pagination work?
a
WIthout knowledge of the specific API I would start removing bits of optional config and see if you get the desired behaviour.
Copy code
pagination_request_style: jsonpath_paginator
      stop_on_empty_next_url: true
      next_page_token_path: $.next
Start with removing these maybe?
k
Ok will try this
@Andy Carter This is LeafLink API response and here’s how the pagination works based on the sample response:
Copy code
{
  "count": 19613,
  "next": "<https://app.leaflink.com/api/v2/orders-received/?limit=50&offset=50>",
  "previous": null,
  "results": [{record1}, {record2}]
}
To fetch all records, we need to keep calling the URL in the 
next
field until it becomes 
null
. Each 
next
URL gives the next batch of 50 records.
Also I tried removing this code but returns only 50 records
Copy code
pagination_request_style: jsonpath_paginator
stop_on_empty_next_url: true
next_page_token_path: $.next
a
Sorry I cannot be more help here but definitely something up with the pagination. I do not see 
stop_on_empty_next_url
as a config setting for the tap so I would remove that at least. Perhaps try just keeping 
next_page_token_path
?
k
Nope removing that also it is not working
Thanks for your help
a
The other thing you can do that is a bit hacky but to edit the python code where your tap is installed to print/log some more messages to the console So that might be editing files in 
.meltano/extractors/tap-rest-api-msdk/lib/py/site-packages/tap-api-rest-msdk
So you might be able to edit some python here: https://github.com/Widen/tap-rest-api-msdk/blob/main/tap_rest_api_msdk/streams.py#L545 to get a bit more insight and see if your 
next
links are correct
k
Right @Andy Carter
Will try that
r
Looking at the tap code, I think you might also want
Copy code
pagination_response_style: hateoas_body
> Note: Under the HATEOAS model, the returned token contains all the required parameters for the subsequent call. The function splits the parameters into Dict key value pairs for subsequent requests. https://github.com/Widen/tap-rest-api-msdk/blob/071b54032beecba75f6100b90281c1d88a47dfde/tap_rest_api_msdk/streams.py#L512-L575 The default implementation does not parse and reapply URL parameters (`offset`/`limit`) from the next page token; it will instead supply the value as its own URL parameter in the next request, e.g. 
?page=https%3A%2F%<http://2Fapp.leaflink.com|2Fapp.leaflink.com>%2Fapi%2Fv2%2Forders-received%2F%3Flimit%3D50%26offset%3D50
🙌 3
k
Thanks @Reuben (Matatika) for helping me
np 1
7 Views
Open in Slack
PreviousNext
Powered by