I am running pipelinewise's tap-postgres and seein...
# troubleshooting
a
I am running pipelinewise's tap-postgres and seeing a 6-minute delay before the first record arrives (from 1 to 2), and then it is only one record (2). After that (3, 4, and 5) things seem to be running as expected. Any ideas why this delay might be happening, or how I can get more information out of the tap?
message has been deleted
Copy code
--log-level=debug
I added this, but the output did not give any additional information about the 6-minute hang
v
Very hard to know as we're getting to postgres specefic stuff. Really with any kind of connection once you ask the server "hey can I get some data" if it takes 6 minutes to respond it's really on the server to tell you why it took 6 minutes. Things to think about:/try 1. Try running that exact query in your own database browser, is it fast? Does it take 6 minutes? Can you get it to take 6 minutes? If you run the tap with the same state does it take 6 minutes over and over? 2. What's the network look like between you and this server? 3. Postgres has some kind of server side debugging you could look into, but I'd think about 1/2 to narrow your options on that side
a
I'd first want to narrow down if this is a SQL-side query performance issue vs something else going on in the tap. My first hunch is that this might be query performance issue. The
order_by
clause in SQL almost always means you need to find all records before you can return any - due to the fact that you can't return any of them until you know which of them is the very smallest.
Can you confirm if you get similar performance issues when running the same query directly in Postgres?
There could be an optimization strategy on the table side (maybe adding an index?) which might optimize the time-to-first row - assuming query performance is actually at issue here.