Hey, I hope I'm reaching out in the right channel. I've created a custom catalog from a tap (based on the existing one) to add a new field. However, in some cases, this field is not returned at all by the REST API — not even as

null

, but completely missing — which leads to a Singer exception. I was wondering if there's a catalog's property designed to handle this kind of situation? I’ve already tried defining the field as nullable and using

additionalProperties

, but I’m still encountering the Singer error when the field is absent from the object. Here is the Singer error :

2025-07-25T10:06:57.048305Z [error  ] Loading failed        code=1 message="singer_sdk.exceptions.InvalidRecord: Record Message Validation Error: {'sub_prop_1': 'abc', 'sub_prop_2': 'def'} is not of type 'string'"

Here is what i have tried in the catalog :

{
  "streams": [
    {
      "tap_stream_id": "obj",
      ...,
      "schema": {
        "properties": {
          "prop_1": {
            "type": ["array", "null"],
            "items": {
              "type": "object",
              "properties": {
                "sub_prop_1": { "type": ["string", "null"] },
                "sub_prop_2": { "type": ["string", "null"] },
                "optional_sub_prop_3": { "type": ["string", "null"] }
              },
              "additionalProperties": true
            }
          }
        }
      }
    }
  ]
}

Thanks in advance to anyone who takes the time to help ☺️

I'm reading

{'sub_prop_1': 'abc', 'sub_prop_2': 'def'} is not of type 'string'

as if you had

"items": {
              "type": "string"
            }

What command are you running?

i am running this command :

meltano el --full-refresh tap-jira target-postgres --catalog tap-jira-catalog.json

i believe the command works, the error occurs after adding the property i need which is :

"optional_sub_prop_3": { "type": ["string", "null"] }

Can you run

meltano invoke tap-jira | grep '{"type":"RECORD"' | grep '"prop_1":'

so we can inspect the format of the raw record?

Here is the output :

"prop_1":[{"sub_prop_1":"<https://google.com>","sub_prop_2":"123","sub_prop_3":"v1.2.3","sub_prop_4":false,"sub_prop_5":true}]

So to confirm, you only get the

is not of type 'string'

error after adding

"optional_sub_prop_3": { "type": ["string", "null"] }

to the schema? I did check this in https://www.jsonschemavalidator.net/ and everything came back fine...

Yes, after adding this to the catalog :`"optional_sub_prop_3": { "type": ["string", "null"] }` . The error occurs when the

optional_sub_prop_3

property is missing from the REST API response. For some reason, it's not always included, but I do need this information when it's present. That's why I was looking for a way to indicate to the catalog that this property may not always be there

I'm not sure what else it could be, unless it's something to do with caching? Your schema looks correct to me. @Edgar Ramírez (Arch.dev) may have a better idea. If you just want to get past the error, you could try setting

validate_records: false

for

target-postgres

and see what gets loaded? https://hub.meltano.com/loaders/target-postgres#validate_records-setting