I'm trying to figure out the best way to handle "custom object" properties that will have arbitrary key and value pairs when using the Meltano SDK. I defined the property in the stream like this

Property('custom_fields', ObjectType(additional_properties=True))

but then when it runs it just complains that all the sub-properties are not found in the catalog and there's no way to include them. Do I have to dynamically retrieve the list of possible keys and include it in the schema, or is there a way to tell the SDK that this is just an object and to store it as an JSON object without a set schema?

by no means sure this is the "right" way, so curious to see what else comes up 😬 but I've sometimes used just

ObjectType()

and set

TYPE_CONFORMANCE_LEVEL = TypeConformanceLevel.ROOT_ONLY

on the stream, dunno if there's a way to alter that at field level though

from singer_sdk.helpers._typing import TypeConformanceLevel

(prepares to alter a bunch of taps if there's a cleaner way 😅 )

This API has an endpoint that describes all custom fields and I already have a stream that reads it (since there's other metadata on the custom fields too) so I was thinking I could use that stream to dynamically make a schema for the respective objects, but then I'm not sure the best way to "read" from that stream as I don't think it would be like a child stream of the custom object definition...

Ahh ok

Turns out the custom field API doesn't actually provide ALL the custom fields (how nice) but

TYPE_CONFORMANCE_LEVEL

works so I guess I have to go with that anyhow. Thanks for that!