Hey everyone, happy tuesday! Has anyone ever introduce error management in their tap? I would love to add tolerance for 500 errors, but am unsure of where to put my try-catch. There 500 errors are coming from an internal API, so sadly are not 100% avoidable, but I would love for my tap to gloss over any instead of crashing. Thanks!

The SDK does have built-in HTTP error handling in RESTStream.validate_response, but I think you’re interested not in retrying after a 500 response but rather just continue to the next stream or partition?

Yes exactly

So, just override default

validate_response

method in your Child stream class? .. probably just wrap it around the actual

validate_response

and catch 500 error and just

pass

. If this doesn't make sense.. probably share your repo? I'll just raise a PR for your reference

catch 500 error and just

pass

@kk I think the problem with that approach is that the stream class would try to parse an invalid response and thus fail down the line. The solution is probably closer to what’s described in #280 or #1689.

But, it would be only failing for that particular child record.. right? This would still continue for the other parent records. I probably am missing something. I am looking into the links you shared

It would actually crash the entire process 😕

right. cause, validating response is one thing. If there's no issue.. we proceed to process the data and then it would fail with some key error or something

Thanks to you both! I'll look into it @edgar_ramirez_mondragon 😄

So, can you try with wrapping sync_records in try except and

pass

if its 500 exception?

@edgar_ramirez_mondragon I see you've been working on this issue with this PR - are there any updates on this front?

@Stéphane Burwash I've just pushed tests in that PR, but I think it's a different issue to the one described in this thread. The PR that address it is https://github.com/meltano/sdk/pull/1754.

Ah I went down to wrong pr rabbit hole then. And this is still in progress? Could be very useful on our side

@Stéphane Burwash the contributed PR is awaiting response and a bit stale, still WIP

😢 oh well, guess I'll have to wait or look into it myself

sorry about that. I will look it again sometime this week.