I'm trying to extract data from

tap-postgres

and load it into

target-postgres

. It works as expected for a bunch of tables but for couple tables that have many rows ~ 80 Millions it fails with the error

This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.OperationalError) connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
INFO - [el-load] 	This probably means the server terminated abnormally
 {pod_manager.py:395} INFO - [el-load-inventory] 	before or while processing the request.

It has extracted and loaded ~ 20 Million rows. I have tried a few other times and it still fails with the same error.

tap-postgres

- MeltanoLabs version 0.0.9

target-postgres

- MeltanoLabs version 0.0.14 Does anyone know what the issue might be? We get the same error for

tap-postgres

and loader

target-bigquery

connection dropped for some reason. Ideally tap-postgres would retry the connection and hook back up on the last server side cursor it had but I'm not sure

Each table has it's own job

you're writing to localhost in production I'd bet there's some optimizations you could do to that postgres service to make things work better. If you really need a large read of a local postgres database that's a bit odd and I'd look into that 🤷 just some quick 2c

The postgres DBs are GCP CloudSQL instances and connecting to those DBs you need a cloudsql-proxy (that's why it looks as local postgres DB) but it's not 🙂