https://linen.dev logo
Join Slack
Powered by
hello, I'm running a relatively long-running `tap-...
# plugins-general
c
hello, I'm running a relatively long-running 
tap-postgres
for the first time... it's connecting to a read-replica hosted in GCP and the target is a Snowflake instance; the tap runs for around 1.5 hrs before failing with the following error:
Copy code
psycopg2.InterfaceError: connection already closed
is this something the postgres replica database is doing? or is it a failure of the tap to reconnect?
I'm using the standard 
tap-postgres
so I assume the transferwise variant, incremental replication on ~120 tables, none bigger than ~9m rows
d
Hey chris, I had this issue and the way i was able to get around it was by messing with the itersize and then the loaders batch_size_rows
I am using snowflake as my loader
I have my itersize set to 500k (though it could be larger probably due to your infra) and my batch_size_rows to 500000 too
c
how do you control the itersize? I don't see it as on of the config options for 
tap-postgres
d
for pipelinewise its there. Sometimes the Hub doesn’t have all the options the tap has https://github.com/transferwise/pipelinewise-tap-postgres
ie
Copy code
name: tap-postgres--staging
            config:
              host: removed
              user: removed
              dbname: removed
              default_replication_method: LOG_BASED
              itersize: 5000000
c
ahh, that's good to know, thanks... I'll try messing with these values
do you think matching the itersize to the batch_size_rows was what solved this?
d
I am not 100% but i suspect it yeah
c
okay I'll give it a shot, thanks for your help! 🤞
Open in Slack
PreviousNext
Powered by