Hi, I was going through the flattening logic in si...
# troubleshooting
p
Hi, I was going through the flattening logic in singer-sdk and came across a problem: the schema (attached below) and a record for that schema (in the thread) are giving error after flattening, on json validation:
Copy code
{
  "attributes": {
    "lastMessageIn": {
      "id": "64c84b5ad14f695ce4d5160b",
      "sentAt": "2023-08-01T00:01:30.659Z",
      "createdAt": "2023-08-01T00:01:30.914Z",
      "meta": {
        "trackingId": "64c83dbeb0bc8f291035157e",
        "to": null
      },
      "channel": "chat"
    }
  }
}
I think there's problem in how null is handled in schema flattening vs record flattening
here
if the schema defines that a field can be array or null then the flattened schema should be
{"type": ["string", "null"]}
Right now I can handle this by using post_process of the stream and replace
null
value to be empty
[]
but I think something is off here
The schema and record are a part from kustomer tap
I can confirm that by adding the following code:
Copy code
elif "array" in v["type"] or "object" in v["type"] and max_level > 0:
                if "null" in v["type"]:
                    items.append((new_key, {"type": ["string", "null"]}))
                else:
                    items.append((new_key, {"type": "string"}))
the stream works.
I've opened an issue, can immediately send a PR if given a green flag 🙂
e
PRs welcome! You may need to update the snapshots and perhaps add a new test case and even a new field with the required=True flag (non-nullable).