My SDK taps started throwing an error over the wee...
# troubleshooting
s
My SDK taps started throwing an error over the weekend that looks like this:
Copy code
meltano install extractor tap-gong


Installing 1 plugins...
Installing extractor 'tap-gong'...
Extractor 'tap-gong' could not be installed: failed to install plugin 'tap-gong'.
  Running command git clone -q <https://github.com/immuta/tap-gong.git> /tmp/pip-req-build-ulyq992e
    ERROR: Command errored out with exit status 1:
     command: /src/data-infrastructure/meltano/.meltano/extractors/tap-gong/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-ut51fk2i/simpleeval_c7a4ae89a7e74b83afeaf64f436b8c75/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ut51fk2i/simpleeval_c7a4ae89a7e
74b83afeaf64f436b8c75/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setupto
ols import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-pip-egg-info-3fq1go2j
         cwd: /tmp/pip-install-ut51fk2i/simpleeval_c7a4ae89a7e74b83afeaf64f436b8c75/
    Complete output (1 lines):
    error in simpleeval setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding <https://files.pythonhosted.org/packages/62/25/aec98426834844b70b7ab24b4cce8655d31e654f58e1fa9861533f5f2af1/simpleeval-0.9.10.tar.gz#>
sha256=692055488c2864637f6c2edb5fa48175978a2a07318009e7cf03c9790ca17bea (from <https://pypi.org/simple/simpleeval/>). Command errored out with exit status
 1: python setup.py egg_info Check the logs for full command output.
ERROR: Cannot install tap-gong because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
Failed to install plugin(s)
The taps are installed in a Docker container built from
meltano/meltano:v1.79.1-python3.8
. I am able to install them on my local machine's environment (i.e. only fails during the dockerized runs.) I am guessing I have done something wrong in the way I set up my tap requirements / pyproject file, but not sure what. Anyone seen this error before?
here, for example, is the pyproject.toml for that tap above: https://github.com/immuta/tap-gong/blob/main/pyproject.toml
More context -- this happened after I incremented my Meltano project dependency from
"^1.78.0"
to
"^1.79"
m
We are facing a similar issue, locally everything is working but only while building the docker container the installation process errors out:
Copy code
Orchestrator 'airflow' could not be installed: failed to install plugin 'airflow'.
    ERROR: Command errored out with exit status 1:
     command: /project/.meltano/orchestrators/airflow/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"'; __file__='"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r8lhwbha
         cwd: /tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding <https://files.pythonhosted.org/packages/d1/a2/9d1fba3287a65f81b9d1c09c4f7cb16f8ea4988b1bc97ffea0d60983338f/Flask-OpenID-1.2.5.tar.gz#sha256=5a8ffe1c8c0ad1cc1f5030e1223ea27f8861ee0215a2a58a528cc61379e5ccab> (from <https://pypi.org/simple/flask-openid/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
The pip_url which we are using is:
Copy code
pip_url: psycopg2==2.9.1 apache-airflow==2.1.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.8.txt>
It suddenly started, also using meltano 1.79.1
Looks like the problem we are facing is due to an upgrade of setuptools: https://github.com/pypa/setuptools/issues/2086
We are not able to pin setuptools ourselves via the pip_url. And it seems that every python2 package no longer works with Meltano due to the upgrade of setuptools. Is it possible to pin setuptools to 57.5.0 in
<https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/venv_service.py#L177>
?
e
@stephen_bailey thanks for reporting. I filed an issue for the SDK-based taps since I noticed the same thing: https://gitlab.com/meltano/sdk/-/issues/213. @martijn_wouters that MR above does seem to fix the issue for all pip plugins, so we'll get a patch out ASAP. I'm surprised no one has logged an issue in the Airflow repo, people will face this issue eventually when they upgrade
setuptools
p
I was wondering why my deploy failed ha, will be following this thread šŸ‘€
a
@edgar_ramirez_mondragon, <!subteam^S02BCD9FFEF> - I think a call šŸ¤™ in 30 would be awesome. It seems like this is hitting the Python community pretty hard and fast. I'd like to better understand what actions (if anything) we can take in the next 24-48 hours. I'll drop a zoom link in #CKHP6G5V4
I'm happy to report we've published a new patch release which pins the
setuptools
version in each Meltano-managed virtual environment. https://meltano.slack.com/archives/CFG3C3C66/p1631054744023100 Huge thanks to everyone who helped debug this, and especially @maarten_van_gijssel for submitting the fix. Note:
The Python 3.9Ā dockerĀ image will be published tomorrow afternoon. (PingĀ #CKHP6G5V4Ā for more info.)
s
Amazing, thanks @maarten_van_gijssel !!