Hi, could someone please help me with an error I’m...
# getting-started
v
Hi, could someone please help me with an error I’m getting using tap-postgres? After setting up extractor (postgres) and loader (also postgres), I’m trying to run it on a user table. It fails with a long description, but seems to be due to this error:
Copy code
CRITICAL ('`schema` is an invalid JSON Schema instance ... ... \'items\': {\'$ref\': \'#/definitions/sdc_recursive_string_array\'}}" is recursive
after farther investigation, it seems that it fails for the tables containing arrays
a
I’ve not seen this particular issue before - are you able to tell if the error is coming from the tap or the target? From the message, it seems (but I don’t want to assume) that the tap may be sending the schema definition with a
$ref
, which the target is not able to properly parse. My first suggestion is generally to try a different fork of the tap or target. Because taps generally should be sending already-resolved schemas to the target, I might suggest trying to swap for another variant of tap and see if that resolves the issue.
A similar issue logged against the target-redshift seems to confirm the above interpretation. You may need to find a tap variant that emits the schema as a simple array of strings (without recursive ref) - or else find a target variant that knows how to process that recursive type. https://github.com/datamill-co/target-redshift/issues/23
v
@aaronsteers thank you for your answer. I tried different implementations of tap-postgres (custom extractor of versions 0.0.61 and the latest 0.2.0, and transferwise). All of them produced the same error. I tried all of them with the same target though.
d
@vladyslav_zavalykhatko Have you tried using the transferwise variant of both tap-postgres and target-postgres? Those should literally be made for each other 🙂
If that works, I think we should list the
$ref
error under https://meltano.com/plugins/loaders/postgres.html#troubleshooting and recommend using the
transferwise
variant instead
v
@douwe_maan that sounds like a very good suggestion! will try soon, thank you
v
@douwe_maan I only now came back to this. the tap/target are set up to be transferwise, but atm no tables are visible for `tap-postgres`:
Copy code
(meltano) [etl@ip-172-31-44-46 prod-db-to-dwh]$ meltano select tap-postgres --list
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        sdfsdfs.*
        ebdb-public-users.*
        users.*

Selected attributes:
is there anything you’d suggest me to try?
but otherwise, I finally made it work using transferwise on both sides. would you want me to help with this?
d
@vladyslav_zavalykhatko You can find the source code for https://meltano.com/plugins/loaders/postgres.html at https://gitlab.com/meltano/meltano/-/blob/master/docs/src/plugins/loaders/postgres.md. You can then edit that file using the Web IDE and submit a change for us to review in a merge request. Have you done that before?
v
not via gitlab webide, but I have some idea 🙂