Hi everyone!
What is the proper way to extract js...
# troubleshootinga
andrio_frizon
11/30/2023, 3:37 PMHi everyone!
What is the proper way to extract jsonb fields from postgres and load on another postgres instance?
Right now my pipeline is breaking unless I override the schema generated, explicitly setting the field type as “object”
andrio_frizon
11/30/2023, 3:38 PMOn my case, I have a field column called
mapping Copy code
"mapping": {}, Copy code
"mapping": {
"type": [
"object"
]
}, Copy code
schema:
public-table_name:
mapping:
type:
- objectandrio_frizon
11/30/2023, 3:39 PMWhen loading on target-jsonl it works fine, so I’d also appreciate an indication on where to investigate it further, I’m not sure if this is a tap-postgres or target-postgres issue
e
edgar_ramirez_mondragon
11/30/2023, 4:17 PMHi Andrio 👋
Which variant of tap-postgres and target-postgres are you using?
a
andrio_frizon
11/30/2023, 4:20 PMHi Edgar!
I’m using meltanolabs variant on both tap and target
e
edgar_ramirez_mondragon
11/30/2023, 6:49 PMOk, I think the issue in the tap is caused by https://github.com/MeltanoLabs/tap-postgres/pull/239.
Right now my pipeline is breaking unless I override the schema generatedIs it a schema validation failure?
a
andrio_frizon
11/30/2023, 7:19 PMI don’t think it is a scheme validation failure, it looks like that extraction worked fine and it broke when target-posgres tried to insert the data:
Copy code
RuntimeError: Neither type nor anyOf are present. Unable to determine type.e
edgar_ramirez_mondragon
11/30/2023, 7:43 PMI see, thanks for sharing. That should've been fixed by https://github.com/MeltanoLabs/target-postgres/pull/183 and shipped in v0.0.8, so my guess is you're on an older version of the target?
a
andrio_frizon
11/30/2023, 8:02 PMYou are correct, the error I shared before was using 0.0.7!
I just updated to 0.0.9 and it raised a different error,
NotImplementedErrore
edgar_ramirez_mondragon
11/30/2023, 9:51 PMI see. Seems this was actually introduced by the PR I mentioned 😅. Quick fix seems to be to update the
NOTYPEa
andrio_frizon
11/30/2023, 9:54 PMThat’s great! I’ll try this as soon as I can!