Hi have been using Meltano for past 2 years Suddenly today g Meltano #troubleshooting

Hi , have been using Meltano for past 2 years. Sud...

sumit_singh

07/04/2024, 11:04 AM

Hi , have been using Meltano for past 2 years. Suddenly today got few failures. Attaching few lines from logs. After sometime it started working normally.

self.chunk_left = int(line, 16)

ValueError: invalid literal for int() with base 16: b''

Copy code

[1m  File "/home/ubuntu/meltano-projects/mysql-snowflake-binlog/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/response.py", line 700, in _update_chunk_length[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.584449Z[0m [[32m[1minfo     [0m] [1m    self.chunk_left = int(line, 16)[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.584571Z[0m [[32m[1minfo     [0m] [1mValueError: invalid literal for int() with base 16: b''[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.584713Z[0m [[32m[1minfo     [0m] [1m                              [0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.584832Z[0m [[32m[1minfo     [0m] [1mDuring handling of the above exception, another exception occurred:[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.584942Z[0m [[32m[1minfo     [0m] [1m                              [0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585046Z[0m [[32m[1minfo     [0m] [1mTraceback (most recent call last):[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585149Z[0m [[32m[1minfo     [0m] [1m  File "/home/ubuntu/meltano-projects/mysql-snowflake-binlog/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/response.py", line 441, in _error_catcher[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585306Z[0m [[32m[1minfo     [0m] [1m    yield                     [0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585435Z[0m [[32m[1minfo     [0m] [1m  File "/home/ubuntu/meltano-projects/mysql-snowflake-binlog/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/response.py", line 767, in read_chunked[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585572Z[0m [[32m[1minfo     [0m] [1m    self._update_chunk_length()[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585680Z[0m [[32m[1minfo     [0m] [1m  File "/home/ubuntu/meltano-projects/mysql-snowflake-binlog/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/snowflake/connector/vendored/urllib3/response.py", line 704, in _update_chunk_length[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585783Z[0m [[32m[1minfo     [0m] [1m    raise InvalidChunkLength(self, line)[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m
[2024-07-04, 16:08:54 IST] {subprocess.py:92} INFO - [2m2024-07-04T10:38:54.585884Z[0m [[32m[1minfo     [0m] [1msnowflake.connector.vendored.urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)[0m [36mcmd_type[0m=[35mloader[0m [36mname[0m=[35mtarget-snowflake[0m [36mrun_id[0m=[35m0d7a5ed2-ba63-4546-9410-d398fee0dd5e[0m [36mstate_id[0m=[35mtap-mysql-payment-grp-to-snowflake[0m [36mstdio[0m=[35mstderr[0m

Edgar Ramírez (Arch.dev)

07/04/2024, 4:52 PM

That's odd. Let me search through the snowflake's GitHub issue trackers in case this is a known problem.

Edgar Ramírez (Arch.dev)

07/04/2024, 6:10 PM

No luck with that 🫤. Is this a transient issue or is it something you've seen consistently in recent days?

sumit_singh

07/05/2024, 7:47 AM

recently, like yesterday ( for many runs) and today ( so far 2 runs)

Edgar Ramírez (Arch.dev)

07/05/2024, 7:32 PM

Gotcha. Do you run

meltano install

as part of your pipeline?

Edgar Ramírez (Arch.dev)

07/05/2024, 7:32 PM

Or is this a long-lived installation of both meltano and the plugins?

sumit_singh

07/12/2024, 10:57 AM

its a long-lived installation of both meltano and the plugins

sumit_singh

07/12/2024, 10:57 AM

did a one time installation

sumit_singh

07/12/2024, 10:59 AM

meltano.yml has schedule and the dag utilizes that

Copy code

schedules:
- name: pipeline-1
  interval: 0/30 * * * *
  extractor: tap-mysql
  loader: target-snowflake
  transform: skip
  start_date: 2023-10-26 09:24:42.930487

sumit_singh

07/12/2024, 11:01 AM

Sharing a command from the airflow logs which is run

Copy code

Running command: ['bash', '-c', 'cd /home/ubuntu/meltano-projects/hedwig; .meltano/run/bin schedule run pipeline-1']

Edgar Ramírez (Arch.dev)

07/12/2024, 3:59 PM

I've never seen this error

Copy code

snowflake.connector.vendored.urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

and I also can't find an issue for something like that in the snowflake connector repo 🫤

sumit_singh

07/12/2024, 6:21 PM

But a general issue can be found on https connection with urllib3/response.py on internet

Edgar Ramírez (Arch.dev)

07/12/2024, 8:10 PM

And what are the usual causes of that exception?

sumit_singh

08/02/2024, 6:38 AM

https://github.com/psf/requests/issues/5583

sumit_singh

08/02/2024, 6:38 AM

The issue didn't occur again

7 Views

Open in Slack

Previous Next