https://linen.dev logo
Join Slack
Powered by
I’m just getting started setting up a MySQL to Sno...
# plugins-general
b
I’m just getting started setting up a MySQL to Snowflake pipeline. The first problem I ran into was in connecting to MySQL. I ran into this error. https://stackoverflow.com/questions/45368336/error-keyerror-255-when-executing-pymysql-connect. The fix here worked for me. Anybody else every run into this? I was planning on creating an issue for this. I know the fix, but I’m not sure how to test or even if anyone else can reproduce the issue. After I got the connection working, I ran into 
pipe closed by peer or os.write(pipe, data) raised exception.
less that 10% of the way through the initial load (185K out of 4.7M). I saw others have run into this type of error when the tap is faster than the loader (I think). Would upping my warehouse size in snowflake help? Would changing the buffer size help?
The first connection error is caused by an old version of pymysql. Here’s the error I got:
Copy code
root@106a7c474d38:/project# meltano invoke tap-nlp-data --discover
time=2021-01-12 23:32:59 name=tap_mysql level=CRITICAL message=255
Traceback (most recent call last):
  File "/project/.meltano/extractors/tap-nlp-data/venv/bin/tap-mysql", line 8, in <module>
    sys.exit(main())
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/tap_mysql/__init__.py", line 404, in main
    raise exc
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/tap_mysql/__init__.py", line 401, in main
    main_impl()
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/tap_mysql/__init__.py", line 384, in main_impl
    log_server_params(mysql_conn)
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/tap_mysql/__init__.py", line 350, in log_server_params
    with connect_with_backoff(mysql_conn) as open_conn:
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/tap_mysql/connection.py", line 28, in connect_with_backoff
    connection.connect()
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/pymysql/connections.py", line 931, in connect
    self._get_server_information()
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1269, in _get_server_information
    self.server_charset = charset_by_id(lang).name
  File "/project/.meltano/extractors/tap-nlp-data/venv/lib/python3.6/site-packages/pymysql/charset.py", line 38, in by_id
    return self._by_id[id]
KeyError: 255
This is fixed in more recent versions of pymysql. Fix was implemented here: https://github.com/PyMySQL/PyMySQL/commit/4b7e9c98c0441449352d732f6a2453e4c868505c
d
@bryan_wise I assume you're using https://github.com/transferwise/pipelinewise-tap-mysql? I suggest filing an issue there to bump the pymysql version past 0.8.0 (which is the first release that has that commit), although it may not be a straightforward dependency bump since they already moved from 0.7.11 to 0.9.3 in version 1.3.1, and then back to 0.7.11 again in 1.3.2 because of backward incompatibility: https://github.com/transferwise/pipelinewise-tap-mysql/blob/e9fd0d2120a5458ba9e1a60d0942681ea3bf3b26/CHANGELOG.md#132-2020-06-15
After I got the connection working, I ran into 
pipe closed by peer or os.write(pipe, data) raised exception.
 less that 10% of the way through the initial load (185K out of 4.7M).  I saw others have run into this type of error when the tap is faster than the loader (I think).  Would upping my warehouse size in snowflake help?  Would changing the buffer size help?
Are you seeing this on the latest version of Meltano? v1.64.0 (https://meltano.slack.com/archives/CP8K1MXAN/p1610064894064300) was supposed to fix this issue (https://gitlab.com/meltano/meltano/-/issues/2478), so if you're still seeing it in a newer version, please file an issue with full log output and stack traces so I can continue debugging.
b
For tap-mysql, yes that is the tap I’m using. I’ll file that issue. Thanks for the guidance. For the other issue. I have not done that upgrade yet. I read about and thought it might help but haven’t had time to try it yet. I will try that tomorrow. Thanks again!
2 Views
Open in Slack
PreviousNext
Powered by