Hi all, I am using tap-s3-csv to load the data fro...
# troubleshooting
s
Hi all, I am using tap-s3-csv to load the data from S3 bucket to Postgresql database. For some reason, I started getting the following error when I tried to load 4 of 200MB files. I have run it successfully to load the 659M file. could anyone know how to fix this issue?
Copy code
2024-03-08T18:11:44.586440Z [info     ] time=2024-03-08 18:11:44 name=tap_s3_csv level=CRITICAL message=("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')) cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.586936Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.587268Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 444, in _error_catcher cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.587609Z [info     ]     yield                      cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.587901Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 567, in read cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.588174Z [info     ]     data = self._fp_read(amt) if not fp_closed else b"" cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.588448Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 533, in _fp_read cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.588711Z [info     ]     return self._fp.read(amt) if amt is not None else self._fp.read() cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.588983Z [info     ]   File "/usr/local/lib/python3.9/http/client.py", line 463, in read cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.589255Z [info     ]     n = self.readinto(b)       cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.589726Z [info     ]   File "/usr/local/lib/python3.9/http/client.py", line 507, in readinto cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.590144Z [info     ]     n = self.fp.readinto(b)    cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.590446Z [info     ]   File "/usr/local/lib/python3.9/socket.py", line 704, in readinto cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.590734Z [info     ]     return self._sock.recv_into(b) cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.591018Z [info     ]   File "/usr/local/lib/python3.9/ssl.py", line 1275, in recv_into cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.591300Z [info     ]     return self.read(nbytes, buffer) cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
2024-03-08T18:11:44.591565Z [info     ]   File "/usr/local/lib/python3.9/ssl.py", line 1133, in read cmd_type=elb consumer=False name=tap-s3-csv-data-provider producer=True stdio=stderr string_id=tap-s3-csv-data-provider
I tried to use the latest boto3 lib (1.34.58), but still failed.
I am running it using Dasger on AWS ECS, and execute meltano command. I just tested it using 200MB file from local and it is working fine.
e
You seem to have run into a random disconnect of the (boto3) client