https://linen.dev logo
Join Slack
Powered by
Hi. I've written a custom tap that outputs some ar...
# getting-started
k
Hi. I've written a custom tap that outputs some arrays of objects. Here's a schema snippet.
Copy code
th.Property(
	"data_files",
	th.ArrayType(
		th.ObjectType(
			th.Property("sequence", th.StringType),
			th.Property("document_url", th.StringType, required=True),
		)
	)
)
The data is being loaded into a PostgreSQL database using target-postgres. The arrays are being mapped as 
ARRAY(JSONB)
, as noted in the README.
Note that while object types are mapped directly to jsonb, array types are mapped to a jsonb array.
How do I configure the loader to map to a 
JSONB
instead of a 
ARRAY(JSONB)
?
e
Using Meltano's schema extra I think you could set it to a generic 
object
type
Copy code
extractors:
- name: tap-custom
  schema:
    some_stream_id:
      data_files:
        type: ["object", "null"]
You might need to turn off schema validation.
➕ 1
k
Thanks! How do I turn off schema validation? I see that the 
Sink
class uses a 
validate_records
config (singer_sdk/sinks/core.py#L199). I tried setting that on a loader config, but I'm getting a validation error.
Copy code
Failed validating 'type' in schema['properties']['data_files']
Actually, the error came from my 
target-jsonl
loader, not 
target-postgres
. So, it does work for loading into a PostgreSQL table. But, it breaks the JSONL plugin.
e
Yeah, target-jsonl doesn't support disabling schema validation unfortunately
3 Views
Open in Slack
PreviousNext
Powered by