https://linen.dev logo
Join Slack
Powered by
I’m running this command: `meltano elt tap-mysql t...
# troubleshooting
j
I’m running this command: 
meltano elt tap-mysql target-postgres --job_id=my-id
I’m using the transferwise variant of target-postgres. I’m getting this error whether I’m using python 3.8 or 3.7: 
Loading failed (1): TypeError: an integer is required (got type bytes)
v
Run this in debug mode 
meltano --log-level=debug elt tap-mysql target-postgres --job_id=my-id
And see if you see anything odd in one of your Integer fields coming from tap-mysql. At the end of the day you need to isolate the field that is failing. That tap/target should make them work regardless.
d
@josh_lloyd Can you share some more of the logs so we can figure out where the error originates? If you scroll up to find the line saying 
target-postgres | TypeError: an integer is required (got type bytes)
, you should see the stacktrace
j
message has been deleted
d
Stacktrace:
Copy code
target-postgres       | Traceback (most recent call last):
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 5, in <module>
target-postgres       |     from target_postgres import main
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/__init__.py", line 13, in <module>
target-postgres       |     from joblib import Parallel, delayed, parallel_backend
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/__init__.py", line 119, in <module>
target-postgres       |     from .parallel import Parallel
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/parallel.py", line 28, in <module>
target-postgres       |     from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 22, in <module>
target-postgres       |     from .executor import get_memmapping_executor
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/executor.py", line 14, in <module>
target-postgres       |     from .externals.loky.reusable_executor import get_reusable_executor
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/externals/loky/__init__.py", line 12, in <module>
target-postgres       |     from .backend.reduction import set_loky_pickler
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/externals/loky/backend/reduction.py", line 125, in <module>
target-postgres       |     from joblib.externals import cloudpickle  # noqa: F401
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/externals/cloudpickle/__init__.py", line 3, in <module>
target-postgres       |     from .cloudpickle import *
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 152, in <module>
target-postgres       |     _cell_set_template_code = _make_cell_set_template_code()
target-postgres       |   File "/Users/jlloyd/Documents/dev/DataTeam/meltano-test/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 133, in _make_cell_set_template_code
target-postgres       |     return types.CodeType(
target-postgres       | TypeError: an integer is required (got type bytes)
So it looks like it's cloudpickle complaining
It looks like you have a version of joblib that is incompatible with Python 3.8
Python 3.8 support was added in 0.14.0, though: https://github.com/joblib/joblib/blob/master/CHANGES.rst#release-0140
You may have ended up with a different version of joblib for some reason, since your stacktrace states that line 152 of 
cloudpickle.py
should read 
_cell_set_template_code = _make_cell_set_template_code()
, but it doesn't in 0.16.0: https://github.com/joblib/joblib/blob/0.16.0/joblib/externals/cloudpickle/cloudpickle.py#L152
What 
pip_url
are you using for target-postgres?
j
Copy code
pipelinewise-target-postgres
d
I wonder if you ended up getting some older version for some reason. Can you check inside the 
.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages
directory and find a dir named 
pipelinewise_target_postgres-<VERSION>.dist-info
?
j
yes 2.1.0 is the version number
d
What version number do you see for joblib?
j
the loader used to be the default variant but I deleted the loader in the meltano.yml file and did a new 
meltano add …
from the cli to add this variant. Could that have caused an issue? I would have used a 
meltano remove loader
command but I didn’t see one in the docs.
d
If you ran 
meltano add
or 
meltano install
again, it would've cleared out and overwritten the original target
But we can try that angle: try deleting `.meltano/loader/target-postgres`and then run 
meltano install loader target-postgres
j
joblib appears to be 0.13.2
I’ll try deleting that now
d
Makes sense, that was the requirement under pipelienwise-target-postgres v2.1.0: https://github.com/transferwise/pipelinewise-target-postgres/blob/v2.1.0/setup.py#L24
But that hasn't been released yet in a new version 😬
You can point your 
pip_url
directly at 
git+<https://github.com/transferwise/pipelinewise-target-postgres.git>
to use the latest, or you can move to Snowflake which may not have this issue, since you were only using Postgres to try things out, right?
Either way I suggest filing an issue in https://github.com/transferwise/pipelinewise-target-postgres/issues asking for a new version to be released
j
well, deleting the 
.meltano/loaders/target-postgres
and then re-adding the loader actually solved the problem …
d
That's surprising
Did you get a different version of joblib now?
j
no, still 0.13.2
but I added the loader with python 3.8 yesterday and this time I added it with python 3.7 … don’t know if that had anything to do with it
d
Aah, probably!
It sounds like 0.13.2 is compatible with 3.7, but only 0.14+ is with 3.8
j
makes sense. Thanks a ton for your help!!!!
d
Glad we figured it out!
Open in Slack
PreviousNext
Powered by