I have run into an issue with two taps this week where the streams are outputting records that have null values in primary key columns. They are primary key columns (

id

, for example), but the source system apparently does not have strict limitations on them, or has special cases where they might be omitted. The tap completes successfully, but there are errors on load due to the null primary keys. I'd like to just remove these from the yielded records, and was wondering if others have suggestions. Right now, I'm just adding a filter into `parse_response`:

def parse_response(self, response: requests.Response) -> Iterable[dict]:
        records = extract_jsonpath(self.records_jsonpath, input=response.json())
        yield from [
            row for row in records
            if all(row.get(k) is not None for k in self.primary_keys)
        ]

But wondering if others have tackled this before, or if this would be a generally useful tap feature?