Hi I am trying to copying from `tap postgres` to `target big Meltano #plugins-general

Hi.. I am trying to copying from `tap-postgres` to...

raghav_nayak

05/20/2021, 10:44 AM

Hi.. I am trying to copying from

tap-postgres

target-bigquery

; and I am getting this error.

Copy code

target-bigquery       | CRITICAL 'type' or 'anyOf' are required fields in property: {}

I tried with another DB, it worked fine without specifying any schema related info. But now, after changing the DB, I am getting the above error. What is the meaning of this? I could not understand whether this fits in my context: https://github.com/adswerve/target-bigquery/issues/12#issuecomment-805352727 The error is thrown from the code

Copy code

.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/schema.py", line 22, in get_type\n    raise ValueError(\n', "ValueError: 'type' or 'anyOf' are required fields in property: {}

dan_ladd

05/20/2021, 12:22 PM

Is there an empty

{}

in your catalog?

raghav_nayak

05/20/2021, 1:18 PM

no.. it is happening to all the tables in that DB

dan_ladd

05/20/2021, 1:30 PM

What the error message is saying is that the target is getting an invalid schema from the tap. It expects either "anyOf" or "type" in the property, but all it's getting is {}. I haven't really used any other targets so I'm not sure how others handle this, but I've always had to fix the catalog and then provide it at runtime.

raghav_nayak

05/20/2021, 5:50 PM

@dan_ladd thanks for the reply. The only difference I see in working or not-working case.. is the schema.. in the working case, the schema is public, and in-the error case, it is not public schema. Does it make any difference?

raghav_nayak

05/20/2021, 5:51 PM

the catalog shows something like this

Copy code

"tap_stream_id": "schema-table_name",
      "schema": {
        "type": "object",
        "properties": {
          "created": {
            "type": [
              "null",
              "string"
            ],
            "format": "date-time"
          },
          "modified": {
            "type": [
              "null",
              "string"
            ],
            "format": "date-time"
          },
          "id": {
            "type": [
              "string"
            ]
          },
          "candidate_id": {
            "type": [
              "null",
              "string"
            ]
          },

raghav_nayak

05/20/2021, 5:59 PM

and how to remove this schema name from the destination as the table name is coming as schema-table_name?

Open in Slack

Previous Next