Hi all! I’m trying to move tables from Postgres to Bigquery using tap-postgres and target-bigquery. Sometimes I run into this error:

line 599, in _helper_simplify
    raise JSONSchemaError('`$ref` path "{}" is recursive'.format(get_ref(root_schema, child_schema['$ref'])))\n', 'target_bigquery.exceptions.JSONSchemaError: `$ref` path "{'type': ['null', 'string', 'array'], 'items': {'$ref': '#/definitions/sdc_recursive_string_array'}}" is recursive\n'] cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

Any idea what to make of it?

Hi. Which version of target bigquery are you using? If it's the default i would suggest to try the transferwise version. So far it's the most stable i've found in terms of handling the schema and schema changes.

I’m using the transferwise one.

I'm using a similar process and never came across that error. Any "weird" field or data type on postgres tables?

yes, probably, I’m suspecting the int4range type, but I don’t know how to avoid that field. I tried specify a mapping with

force_field

in a

table_config

but to no avail. Do you know of a way to clearly log what field it is currently crashing on?

Hi @marten_nilsson, no not really. I've seen some discussions here around tweaking the schema, but never tried myself as i never had a similar issue.

It seems this has been up long before I came here: https://meltano.slack.com/archives/C01TCRBBJD7/p1621970218073800

Yes indeed. Not sure if it would solve the issue, but recently i had an issue when creating a clustered table in BigQuery and the solution was to create the table "manually" on Bigquery and that solved the issue for now.