Hey Everyone has anyone run into the issue of `failed = inst Meltano #troubleshooting

Hey Everyone, has anyone run into the issue of `fa...

Luke

08/27/2024, 6:42 PM

Hey Everyone, has anyone run into the issue of

failed = instance % dB

? followed by

decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>]

I have a table in mysql that I'm trying to move to snowflake via

meltano el tap-mysql target-snowflake

Copy code

+----------------------------+----------------+------+-----+---------+-------+
| Field                      | Type           | Null | Key | Default | Extra |
+----------------------------+----------------+------+-----+---------+-------+
| transaction_reference_id   | bigint(20)     | NO   | PRI | NULL    |       |
| oms_id                     | int(11)        | NO   | PRI | NULL    |       |
| account_id                 | int(11)        | NO   | PRI | NULL    |       |
| transaction_reference_type | tinyint(4)     | NO   | PRI | NULL    |       |
| product_id                 | int(11)        | NO   |     | NULL    |       |
| hold_amount                | decimal(60,30) | NO   |     | NULL    |       |
| entered_on_ticks           | bigint(20)     | NO   |     | NULL    |       |
| active_hold                | tinyint(1)     | NO   | MUL | NULL    |       |
+----------------------------+----------------+------+-----+---------+-------+

and the issue must be the

decimal(60,30)

because if I do a test where I convert that to

TEXT

it works fine does anyone have a solution or workaround for this? I'm fairly new to meltano so sorry if this is a obvious solve UPDATE: appears its not just limited to the

target-snwoflake

plugin as

meltano run tap-mysql target-jsonl

also fails with the same error message, I've attached the full log output in the thread

Luke

08/27/2024, 7:47 PM

Copy code

2024-08-27T19:46:27.339360Z [info     ] Environment 'dev' is active
2024-08-27T19:46:27.744460Z [warning  ] No state was found, complete import.
2024-08-27T19:46:28.091890Z [info     ] time=2024-08-27 19:46:28 name=tap_mysql level=INFO message=Server Parameters: version: 10.11.8-MariaDB-0ubuntu0.24.04.1-log, wait_timeout: 28800, innodb_lock_wait_timeout: 3600, max_allowed_packet: 16777216, interactive_timeout: 28800 cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.093344Z [info     ] time=2024-08-27 19:46:28 name=tap_mysql level=INFO message=Server SSL Parameters(blank means SSL is not active): [ssl_version: ], [ssl_cipher: ] cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.100436Z [info     ] time=2024-08-27 19:46:28 name=tap_mysql level=INFO message=Beginning sync for InnoDB table test.acc_h2 cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.100789Z [info     ] time=2024-08-27 19:46:28 name=tap_mysql level=INFO message=Stream test-acc_h2 is using full table replication cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.102304Z [info     ] WARNING Unknown message type ACTIVATE_VERSION in message {'type': 'ACTIVATE_VERSION', 'stream': 'test-acc_h2', 'version': 1724787988100} cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.104840Z [info     ] time=2024-08-27 19:46:28 name=tap_mysql level=INFO message=Running SELECT `entered_on_ticks`,`transaction_reference_type`,`hold_amount`,`transaction_reference_id`,`product_id`,`account_id`,`active_hold`,`oms_id` FROM `test`.`acc_h2` cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.110693Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.110972Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module> cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111106Z [info     ]     sys.exit(main())           cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111237Z [info     ]              ^^^^^^            cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111352Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/target_jsonl.py", line 92, in main cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111467Z [info     ]     state = persist_messages(  cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111577Z [info     ]             ^^^^^^^^^^^^^^^^^  cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111698Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/target_jsonl.py", line 54, in persist_messages cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111813Z [info     ]     validators[o['stream']].validate((o['record'])) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.111925Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 129, in validate cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112036Z [info     ]     for error in self.iter_errors(*args, **kwargs): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112147Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 105, in iter_errors cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112274Z [info     ]     for error in errors:       cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112386Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/_validators.py", line 300, in properties_draft4 cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112498Z [info     ]     for error in validator.descend( cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112605Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 121, in descend cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112714Z [info     ]     for error in self.iter_errors(instance, schema): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112823Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 105, in iter_errors cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.112933Z [info     ]     for error in errors:       cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.113038Z [info     ]   File "/home/ubuntu/m/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/_validators.py", line 127, in multipleOf cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.113148Z [info     ]     failed = instance % dB     cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.113267Z [info     ]              ~~~~~~~~~^~~~     cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.113376Z [info     ] decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>] cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-jsonl name=target-jsonl producer=False run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=target-jsonl
2024-08-27T19:46:28.127301Z [info     ] time=2024-08-27 19:46:28 name=singer level=INFO message=METRIC: {"type": "counter", "metric": "record_count", "value": 300, "tags": {"database": "test", "table": "acc_h2"}} cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.127643Z [info     ] time=2024-08-27 19:46:28 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "job_duration", "value": 0.02853107452392578, "tags": {"job_type": "sync_table", "database": "test", "table": "acc_h2", "status": "succeeded"}} cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-jsonl name=tap-mysql producer=True run_id=12222b82-a90b-477b-a127-9911eddea56c stdio=stderr string_id=tap-mysql
2024-08-27T19:46:28.147090Z [error    ] Loader failed
2024-08-27T19:46:28.147412Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed

visch

08/27/2024, 8:20 PM

Sounds like a tap-mysql bug? I'm not certain though that is a pretty large decimal number

100 000 000 000 000 000 000 000 000 000

is pretty big 😄

Luke

08/27/2024, 8:21 PM

yeah its the wonderful world of crypto lol

😂 1

😅 1

Luke

08/27/2024, 8:21 PM

if I use

invoke

it appears to run just fine, just not when passing to to a extractor

visch

08/27/2024, 8:23 PM

When you call a loader they are doing some kind of validation on the data which is why you're hitting the issue. most can turn validation off but that's not normally what you want to be doing. This smells like an issue with the size of the decimal being allowed though

Luke

08/27/2024, 8:23 PM

any idea on how to get around it?

Edgar Ramírez (Arch.dev)

08/27/2024, 11:02 PM

Which variant of tap-mysql is this?

Edgar Ramírez (Arch.dev)

08/27/2024, 11:11 PM

Found https://github.com/python-jsonschema/jsonschema/issues/701 which actually mentions Singer, so I'm feeling a déjà vu. Any workaround I can think of involves either patching the schema to avoid validation of

multipleOf

or turning off validation on the target side. • MeltanoLabs/target-snowflake should accept a

validate_records

config. Setting it to

false

is worth a try. (Maybe we need to document it). • If that doesn't work, maybe using schema:

Copy code

plugins:
  extractors:
  - name: tap-mysql
    schema:
      my_table:
        my_column:
          multipleOf: null

mayank_sardana

09/23/2024, 5:44 AM

I am facing this exact same issue with

tap-oracle

and I was able to get this to work by setting schema

validate_records

to false. What are the downsides of this? And what can be a permanent fix? I am happy to open an issue but not sure whether it's a tap issue or a target.

Edgar Ramírez (Arch.dev)

09/23/2024, 5:08 PM

Hi @mayank_sardana.

What are the downsides of this?

If you're using a non-database target, there's a slight risk of loading non-conforming data if the tap sends invalid records (e.g. loading a string in a number field).

And what can be a permanent fix? I am happy to open an issue but not sure whether it's a tap issue or a target.

One thing that might help is knowing the decimal and

multipleOf

values that are causing problems, and see if there's a way to make the tap emit a working pair. If there's nothing the tap can do better, then it's definitely a target issue. We could like at allowing users to control the use of

use_decimal=False

when deserializing JSON.

👍 1

6 Views

Open in Slack

Previous Next