https://linen.dev logo
Join Slack
Powered by
Hi, I have had some success with using the bonobos...
# troubleshooting
h
Hi, I have had some success with using the bonobos variant of the sftp tap, but when moving from my designated dev FTP server to the actual FTP server we want to read from, I encountered a peculiar error. From the logs it appears it connects successfully many times in a row, until it gets an error: “ERROR Exception: Error reading SSH protocol banner”. Googling so far suggests this could be because the server receives too many requests - which may be because of all the times it connected. Has anyone seen anything similar? I am able to connect to the server both using an FTP client and paramiko directly in a python script.
The salient part of the logs/errors:
Copy code
INFO Creating new connection to SFTP...
INFO Connected (version 2.0, client <XXXXX name>)
INFO Auth banner: b'<XXXXX> SFTP Service\r\n'
INFO Authentication (password) successful!
INFO [chan 0] Opened sftp connection (server version 3)
INFO Connection successful
INFO Creating new connection to SFTP...
INFO Connected (version 2.0, client <XXXXX name>)
INFO Auth banner: b'<XXXXX> SFTP Service\r\n'
INFO Authentication (password) successful!
INFO [chan 0] Opened sftp connection (server version 3)
INFO Connection successful
INFO Creating new connection to SFTP...
ERROR Exception: Error reading SSH protocol banner
ERROR Traceback (most recent call last):
ERROR   File "/workspaces/meltano_bonobo/el/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/paramiko/transport.py", line 2211, in _check_banner
ERROR     buf = self.packetizer.readline(timeout)
ERROR   File "/workspaces/meltano_bonobo/el/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/paramiko/packet.py", line 380, in readline
ERROR     buf += self._read_timeout(timeout)
ERROR   File "/workspaces/meltano_bonobo/el/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/paramiko/packet.py", line 609, in _read_timeout
ERROR     raise EOFError()
ERROR EOFError ``````
p
@Henning Holgersen I actually used to work at bonobos and on that tap! Before I left we did see that issue come up a few times but we synced small amounts of data pretty infrequently so it wasnt a problem for us. I suspect theres a bug with the way the connection is being created, its like its reconnecting instead of reusing the connection. Probably a simple bug fix somewhere. Theres also tap-spreadsheets-anywhere that does SFTP but I havent personally used it for that
h
Thank you. It's good to hear that I was on the right track, and that others have seen the same issue. I will take another look at the code, although I doubt I will be able to improve it.
@pat_nadolny I noticed just now that you were the last contributor to that repo more than a year ago! I have been able to rewrite the connector to reuse the connection, reducing (in my testcase) the number of connections made from 60 to 1. Also, our actual source now works 😄 I’m not entirely sure where to go from here, developing taps is new to me, and although it is working I had to disable a test. Since the repo looks dormant I would be hesitant to do a PR in any case, but adding an almost identical variant seems excessive. Any thoughts? ps: The problem with the spreadsheets-anywhere tap is that the password ends up as plaintext both in meltano.yml and in the target table.
p
@Henning Holgersen awesome to hear you got it working! Yeah I used that at a previous job, as far as I know they still use it. Meltano's view on connector ownership and maintenance was highlighted in this blog post related to meltano labs if you wanted to check it out. Of the 4 options numbers 3/4 would make sense for your situation: 1. Single Named Owner 2. Vendor Self-Managed 3. Benevolent Community Member (your personal or organization's repo) 4. Community-Managed Fork with Community Maintainers (MeltanoLabs) We can make your fork with the bug fixes into the default variant listed on meltano hub and you'd become the new active maintainer/variant that we'd direct users to. Or you could fork your updated version to meltano labs where you can start as the primary maintainer but have the option to bring in a new maintainer in the future if you no longer have capacity to support it. Both give you the benefit of community support and contribution but its a preference of the type of ownership you want over it. Can you share the link to your updated fork? Let us know what you think!
h
Wow, thanks, this looks exciting. My fork, and working(-ish) branch, is here: https://github.com/radbrry/tap-sftp/tree/reuseconnection I temporarily renamed the package tap-nicesftp, but that was just as a precaution so that I wouldn’t inadvertently use the original version. My initial thought is a community managed fork. We are able to support the tap for a while now, but eventually the tap will do the things we want to do and we will need to spend our time elsewhere. I will discuss this with my team and get back to you.
@pat_nadolny: I talked with the rest of my team, we are indeed a little pragmatic and think it would be a better long-term solution if the connector was managed by meltanolabs. That being said, we are happy to be the lead for now. I have fixed the unittest errors so it passes the tox tests (with no guarantees for the quality of the fix), and I have tested it with meltano once more. But it would be great with another set of eyes. I have not yet merged the dev branch: https://github.com/radbrry/tap-sftp/tree/reuseconnection
Open in Slack
PreviousNext
Powered by