Getting this error in tap-postgres

psycopg2.errors.QueryCanceled: canceling statement due to statement timeout

I do not see any variable to set to increase statement_timeout. Any help?

@irfan_alam https://github.com/transferwise/pipelinewise-tap-postgres doesn't currently have a way to set a

statement_timeout

, but I think it would be pretty easy to add: The

cfg

dict in https://github.com/transferwise/pipelinewise-tap-postgres/blob/master/tap_postgres/db.py#L42 would need a new

options

key with a value like

-c statement_timeout=100s

. To add a new

statement_timeout

setting, the code would look something like this:

statement_timeout = conn_config.get('statement_timeout')
if statement_timeout is not None:
  cfg['options'] = f'-c statement_timeout={statement_timeout}s'

Are you comfortable forking the repo, making the change, and contributing it back to the main repo? To use your own fork in Meltano , see http://meltano.com/docs/plugin-management.html#using-a-custom-fork-of-a-plugin.

@irfan_alam Great! Did you confirm it works on your system? Note that with the current code, a value of

0

(to disable the timeout completely) would be ignored, since

if conn_config.get('statement_timeout')

would evaluate to

if false

, since

0

is falsey!

updated the PR according to your provided code

@irfan_alam The code looks good to me, but I'm not a maintainer of that repository so I can't approve or merge it. @peter_kosztolanyi Can you or someone on your team please review that PR? 🙂

👋 I think we can merge this but can we try making it more generic by adding an optional

options

key to the config? config.json:

{
  "host": "x",
  "port": "x",
  "user": "x",
  "password": "x",
  "dbname": "x",
  "options": [
    "statement_timeout=60",
    "another_option=120",
    "and_another_option=180"
  ]
}

And in the code something like this:

cfg['options'] = ' '.join(map('-c {0}'.format, conn_config.get('options',[])))

It’s maybe more flexible, wdyt?

I like it!