Hey everyone Anyone had issues running the `target postgres` Meltano #plugins-general

Hey everyone! Anyone had issues running the `targe...

ricky_renner

03/16/2021, 9:00 PM

Hey everyone! Anyone had issues running the

target-postgres

target from Transferwise with this error:

Copy code

target-postgres           | time=2021-03-16 20:47:25 name=target_postgres level=INFO message=Table '"machine_rate_full"' does not exist. Creating... CREATE TABLE IF NOT EXISTS excel_source_files_meltano."machine_rate_full" ("" character varying, "_sdc_batched_at" timestamp without time zone, "_sdc_deleted_at" character varying, "_sdc_extracted_at" timestamp without time zone, "_smart_source_bucket" character varying, "_smart_source_file" character varying, "_smart_source_lineno" numeric, "fromdate" character varying, "machine_hour_rate" double precision, "press_number" character varying, "todate" character varying, PRIMARY KEY ("press_number", "fromdate"))
target-postgres           | Traceback (most recent call last):
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module>
target-postgres           |     sys.exit(main())
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 373, in main
target-postgres           |     persist_lines(config, singer_messages)
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 219, in persist_lines
target-postgres           |     stream_to_sync[stream].sync_table()
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/db_sync.py", line 589, in sync_table
target-postgres           |     self.query(query)
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/db_sync.py", line 313, in query
target-postgres           |     params
target-postgres           |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/psycopg2/extras.py", line 143, in execute
target-postgres           |     return super(DictCursor, self).execute(query, vars)
target-postgres           | psycopg2.errors.SyntaxError: zero-length delimited identifier at or near """"
target-postgres           | LINE 1: ...S excel_source_files_meltano."machine_rate_full" ("" charact...
target-postgres           |                                                              ^
target-postgres           |

Looks like it is detecting an empty column at the beginning of my Excel file that I am pulling over from

tap-spreadsheets-anywhere

, but the Excel file does not have any empty spaces in the column header row.

ricky_renner

03/16/2021, 9:16 PM

Yeah it actually looks like the schema from

tap-spreadsheets-anywhere

is outputing with a column that is an empty name:

Copy code

{
  "type": "SCHEMA",
  "stream": "machine_rate_full",
  "schema": {
    "properties": {
      "press_number": {
        "type": [
          "null",
          "string"
        ]
      },
      "machine_hour_rate": {
        "type": [
          "null",
          "number"
        ]
      },
      "fromdate": {
        "type": [
          "null",
          "string"
        ]
      },
      "todate": {
        "type": [
          "null",
          "string"
        ]
      },
      "": {
        "type": [
          "null",
          "string"
        ]
      },
      "_smart_source_bucket": {
        "type": "string"
      },
      "_smart_source_file": {
        "type": "string"
      },
      "_smart_source_lineno": {
        "type": "integer"
      }
    },
    "selected": true,
    "type": "object"
  },
  "key_properties": [
    "press_number",
    "fromdate"
  ]
}

ricky_renner

03/16/2021, 9:17 PM

@eric_simmerman any idea how this might've happened? I don't see anything wrong with the Excel file

eric_simmerman

03/16/2021, 9:19 PM

This is a xlsx yes?

eric_simmerman

03/16/2021, 9:20 PM

Either way - I haven’t seen it before. Test case or at least a sample of the source file would be a helpful add to the test suite.

eric_simmerman

03/16/2021, 9:22 PM

But at first look - it seems like a technically valid possible parsing.

ricky_renner

03/16/2021, 9:37 PM

Yeah I can send over an example

ricky_renner

03/16/2021, 9:43 PM

Actually... I just "deleted" anything in the surrounding area around the actual data and that fixed the issue

ricky_renner

03/16/2021, 9:43 PM

So there must've been a hanging space character or something

ricky_renner

03/16/2021, 9:48 PM

Except I don't think the

field_names

section was working. I was trying to just define the actual column names, but it still was picking up an empty string for a column... or am I misunderstanding how the

field_names

is supposed to work?

eric_simmerman

03/16/2021, 11:24 PM

field_names simply replaces use of the header row for defining column names - but I don’t believe support has been implemented for Excel. No reason it can’t be - should be straight forward.

13 Views

Open in Slack

Previous Next