I am getting this error -

psycopg2.OperationalError: SSL SYSCALL error: EOF detected

While transferring data from tap-postgres to target-postgres. The table has about 32 Lac rows.

meltano     | Running extract & load...
meltano     | No state was found, complete import.
target-postgres | INFO PostgresTarget created with established connection: `user=xxx password=xxx dbname=xxx host=xxx port=xxx sslmode=prefer sslcert=~/.postgresql/postgresql.crt sslkey=~/.postgresql/postgresql.key sslrootcert=~/.postgresql/root.crt sslcrl=~/.postgresql/root.crl`, PostgreSQL schema: `xxx_schema`
tap-postgres  | time=2021-01-05 19:16:58 name=tap_postgres level=INFO message=Selected streams: [] 
target-postgres | INFO Sending version information to <http://singer.io|singer.io>. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
tap-postgres  | time=2021-01-05 19:25:57 name=tap_postgres level=INFO message=No streams marked as currently_syncing in state file
target-postgres | Traceback (most recent call last):
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module>
target-postgres |   sys.exit(cli())
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 45, in cli
target-postgres |   main(args.config)
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 39, in main
target-postgres |   target_tools.main(postgres_target)
target-postgres | psycopg2.OperationalError: SSL SYSCALL error: EOF detected
target-postgres | 
meltano     | Loading failed (1): (see above)
meltano     | ELT could not be completed: Target failed

Can anyone please help?

Do you have data flowing in from your tap?

No

I am getting this error - 

psycopg2.OperationalError: SSL SYSCALL error: EOF detected

While transferring data from tap-postgres to target-postgres.

@irfan_alam https://github.com/psycopg/psycopg2/issues/533 suggests that this means that the database is closing the connection prematurely, possibly because of an out-of-memory issue. Is that possible?

Well most of the time it failed when configurations was in meltano.yml, but I think when it succeeded 1 time then there were no difference in output

@irfan_alam OK, then I think it's not related to where the config is stored, since Meltano should pass it to the plugin the same way anyway. It's more likely to have been a coincidence, with the real issue having to do with database memory/resource usage.

This happened with I used tap-postgres

psycopg2.OperationalError: SSL SYSCALL error: EOF detected

target-postgres | CRITICAL cursor already closed

Parts of the log

tap-freshdesk  | INFO Completed sync
target-postgres | DEBUG MillisLoggingConnection: 0 millis spent executing: b'BEGIN;'
target-postgres | DEBUG MillisLoggingConnection: 0 millis spent executing: b'BEGIN;'
target-postgres | CRITICAL cursor already closed
target-postgres | Traceback (most recent call last):
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/postgres.py", line 234, in write_batch
target-postgres |   cur.execute('BEGIN;')
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/postgres.py", line 65, in execute
target-postgres |   return super(_MillisLoggingCursor, self).execute(query, vars)
target-postgres |  File "/project/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/psycopg2/extras.py", line 461, in execute
target-postgres |   return super(LoggingCursor, self).execute(query, vars)
target-postgres | psycopg2.OperationalError: SSL SYSCALL error: EOF detected

@irfan_alam It looks your Postgres DB is closing the connection for some reason (possibly because it's running out of memory or other resources), and target-postgres is not handling the error and attempting to reconnect. Do you have any idea why the connection may be getting dropped? You may want to try one of the other target-postgres variants: https://meltano.com/plugins/loaders/postgres.html#alternative-variants One of them may handle dropped connections better.

The tap-freshdesk takes too much time to fetch data, I think more than 20-30 minutes. By the time the Postgres connection will be closed anyhow. Why do you establish a connection beforehand when extracting data? Why don't after tap data are extracted?

@irfan_alam Is the tap waiting for all the data to be extracted before it starts sending records to the target? The tap and target run in a parallel and the tap's stdout output is piped into the target's stdin, so the target shouldn't need to wait until all data is extracted, and typically starts inserting new records just a few seconds after it makes the connection to the DB, when the tap outputs its first record message. If the target is sitting idle for 20-30 minutes and letting the connection expire, the tap may be misbehaving.