It took me less than two hours to make a new REST-tap today, which really shows the usefulness of the SDK. But I stumbled across an issue I thought I had figured out: arbitrary json-fields. On my previous REST-tap I was able to handle arbitrary JSON fields in schemas with

ObjectType()

, but this tap returns empty dicts (

{}

) for some reason. This is the case both for straight object-types and object-arrays. Before I spend hours to figure this out, are there any common issues I should check out?

Is the empty output dependent on the target you're loading data to? I ran into a similar issue when loading to bigquery, but not to jsonl

Good point. This is writing to jsonl, I ran in debug mode a few times as well, the attribute is indeed just an empty dict.

@Henning Holgersen might be caused by changes introduced in https://github.com/meltano/sdk/pull/887. Are you seeing warning messages like

Properties ... were present in the '...' stream but not found in catalog schema. Ignoring.

?

is the best way to get these arbitrary json fields back by downgrading the sdk or is there a better/ recommended way?

@gary_james There is a nice property you can set in the stream:

TYPE_CONFORMANCE_LEVEL = TypeConformanceLevel.ROOT_ONLY

You need to import

TypeConformanceLevel

too:

from singer_sdk.helpers._typing import TypeConformanceLevel

. I have started pondering how this should be documented so that people will find it.

Amazing, thank you!!