Hello, Working on a tap based on a REST API. The tap hits several endpoints and I know that one of them will occasionally fail with a 403 error. However, this endpoint doesn’t have particularly important data and I want the tap to run and return data from all of the other endpoints even if the one fails. My current solution is to add an additional

optional

parameter and override the validate_response method in my stream class (which subclasses RestStream) with

def validate_response(self, response: requests.Response) -> None:
        """Updating validation to handle special case of 403"""
        if self.optional and response.status_code == 403:
            self.logger.warning(
                f"403 Forbidden Error at stream: {self.name}. "
                f"Continuing because stream is marked as optional."
            )
        else:
            super().validate_response(response)

If this is triggered, my target will receive no records and then write an empty file. Mainly curious if anyone else has encountered this and if it could/should be built in to the singer sdk.

https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/client.py#L92-L97 Yeah I did the same. I think you want to be really specific about the error and when it is expected

Well I’d want it to be a warning e.g. “IgnorableAPIErrorWarning”.

Yeah, this is definitely similar to what I want. Although ideally I also want to overwrite as little of the singer sdk as possible if I have to reuse this logic.

@steven_litvack-winkler - Is that an incremental stream, and what behavior would you want for it when a

403

occurs? Do you want to basically just treat it as the end of steam (to be resumed on next run), or sleep and retry, or skip to the next record, or something else?

No it’s not an incremental stream. Yeah I want to treat it as the end of the stream and return no data. The issue is that I have several streams in one tap. So the default behavior is that Meltano will fail and any streams scheduled for after will not run. Hence the work around is to bypass the normal error handling on a FatalAPIError.

Relevant feature proposal: https://github.com/meltano/sdk/issues/280#issuecomment-1237645534

Yeah, I think this is another possible solution. Or I’m thinking as an intermediate it could raise an IgnorableAPIError in validate_response as Derek suggested and then chain an EndOfStreamError in get_records.