Hey anyone experience an issue where the Zendesk extractor fails with a 502 error? In this case the exact error was

Extraction failed (1): requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: <https://nymag.zendesk.com/api/v2/tickets/214/comments.json>

but it’s not always the same ticket ID or endpoint. The extractor starts running smoothly and the first few API calls succeed but less than a minute in one will fail. The URL in question fetches a response successfully in my browser.

This sounds like a rate-limiting error and is probably based on how Zendesk has their API setup. I’m not super familiar with the zendesk tap, but perhaps there’s a way to slow down the requests?

It seems Meltano uses the singer Zendesk tap, which in turn uses the Zenpy library. The library claims to handle any rate limiting with the Zendesk API, so that shouldn’t be an issue?

https://developer.zendesk.com/rest_api/docs/support/usage_limits suggests that 502 is not a rate limiting error:

If the rate limit is exceeded, the API responds with a HTTP 429 Too Many Requests response code and a body that details the reason for the rate limiter kicking in.

Yeah in 3 attempts I saw it twice on

tickets/<id>/comments

and once on

tickets/<id>/metrics

There was a service incident ~12 hours ago https://support.zendesk.com/hc/en-us/articles/1260802729689-Service-Incident-February-19th-2021

We have a custom pipeline to extract data from Zendesk that also uses Zenpy and hits some of the same endpoints that ran successfully twice today with no trouble 🤔

That's very odd 😕

I’m trying now with a

--select tickets.*

throw in to see if that works and we’re now two minutes in, which is further than I got with previous tries

😕

that error was actually on the

ticket_comments

stream. i think

tap-zendesk

fetches metrics and comments on a per ticket basis so it would fetch a ticket, fetch it’s comments and metrics, then move on to the next. so i think it had synced some rows for the base tickets stream but i’m not sure.