Back with more inscrutable errors I ve got a two column SQL Meltano #getting-started

Back with more inscrutable errors! I've got a two-...

simon_podhajsky

05/01/2024, 4:18 AM

Back with more inscrutable errors! I've got a two-column SQL table that doesn't appear to be special in any way, except that both columns are primary keys. That doesn't seem to be a problem yet -

tap-mysql

reads it,

target-duckdb

creates it, all good. Except when the time comes to load the temporary table into the target table,

target-duckdb

chokes for some reason, throwing a

duckdb.duckdb.ParserException: Parser Error: syntax error at or near "FROM"

and leaving behind the temp table in

main

schema. (I'll attach the full traceback in 🧵.) Is this a pattern anyone has seen?

simon_podhajsky

05/01/2024, 4:18 AM

(I'll try to see if killing the double-primary-key-ness of the origin table does anything in the meantime.)

simon_podhajsky

05/01/2024, 4:19 AM

One thing to note below is that the initial INFO line has the correct number of rows from the temp table, meaning that it successfully reads that from the temp table but not its actual contents? Weird.

Copy code

meltano-1  | 2024-05-01T04:04:31.935825Z [info     ] time=2024-05-01 04:04:31 name=target_duckdb level=INFO message=Loading 222 rows into 'adk_wrapped_full.debatovanicz."soutez_teze"' cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.938590Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.938742Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/bin/target-duckdb", line 8, in <module> cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.938902Z [info     ]     sys.exit(main())           cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939013Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/__init__.py", line 427, in main cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939113Z [info     ]     persist_lines(connection, config, singer_messages) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939249Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/__init__.py", line 314, in persist_lines cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939368Z [info     ]     flushed_state = flush_streams( cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939516Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/__init__.py", line 351, in flush_streams cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939660Z [info     ]     load_stream_batch(         cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939792Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/__init__.py", line 392, in load_stream_batch cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.939907Z [info     ]     flush_records(stream, records_to_load, row_count[stream], db_sync, temp_dir) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.940020Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/__init__.py", line 409, in flush_records cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.940107Z [info     ]     db_sync.load_rows(records_to_load.values(), row_count, temp_dir) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.940236Z [info     ]   File "/project/.meltano/loaders/target-duckdb/venv/lib/python3.9/site-packages/target_duckdb/db_sync.py", line 390, in load_rows cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.940352Z [info     ]     cur.execute(self.update_from_temp_table(temp_table)) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb
meltano-1  | 2024-05-01T04:04:31.940442Z [info     ] duckdb.duckdb.ParserException: Parser Error: syntax error at or near "FROM" cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-duckdb name=target-duckdb producer=False run_id=b3d7d89d-612d-48c5-9e09-e9c1b8ce180d stdio=stderr string_id=target-duckdb

simon_podhajsky

05/01/2024, 4:25 AM

Indeed, removing the primary keys from the origin table fixes the issue, but why I could not possibly say.

Edgar Ramírez (Arch.dev)

05/02/2024, 7:30 PM

The way the sql statement is generated seems likely to fail if some assumptions are violated: https://github.com/jwills/target-duckdb/blob/c96527c17bd377d370f3433a4c98c9fc08ae5e19/target_duckdb/db_sync.py#L419-L432

Edgar Ramírez (Arch.dev)

05/02/2024, 7:30 PM

I would have to debug to tell exactly what's the invalid sql being generated, but it's probably worth creating an issue in their repo

simon_podhajsky

05/02/2024, 7:42 PM

Already filed, now updated (https://github.com/jwills/target-duckdb/issues/36) - thanks! If my reading is correct, this might be failing because there are no non-primary-key columns to write.

7 Views

Open in Slack

Previous Next