https://linen.dev logo
Join Slack
Powered by
I'm running Ubuntu 20.04 so by default have Python...
# announcements
t
I'm running Ubuntu 20.04 so by default have Python 3.8 locally. I was originally running a 3.8 venv and was fine with the Gitlab tap and JSONL target but when I started trying to use other taps and targets I ran into issues with dependencies with things being too old to support Python3.8 (postgres and parquet needing newer versions of psycopg2 and pyarrow respectively) so decided to blow away my venv and use pyenv to give me Python 3.7 to solve other dependency issues. Unfortunately I now get errors when running `meltano install`:
Copy code
Traceback (most recent call last):
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 338, in dbapi
    from pysqlite2 import dbapi2 as sqlite
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tom/git/meltano-projects/.venv/bin/meltano", line 11, in <module>
    load_entry_point('meltano==1.52.0', 'console_scripts', 'meltano')()
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/meltano/cli/__init__.py", line 38, in main
    cli(obj={"project": None})
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/meltano/cli/params.py", line 50, in decorate
    project, settings_service.get("database_uri"), default=True
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/db.py", line 38, in project_engine
    engine = create_engine(engine_uri)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 443, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 343, in dbapi
    raise e
  File "/home/tom/git/meltano-projects/.venv/lib/python3.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 341, in dbapi
    from sqlite3 import dbapi2 as sqlite  # try 2.5+ stdlib name.
  File "/home/tom/.pyenv/versions/3.7.9/lib/python3.7/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/home/tom/.pyenv/versions/3.7.9/lib/python3.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
I saw this answer on SO https://stackoverflow.com/a/48229955/2291321 but it doesn't seem to help me. Anyone else had this issue before?
a
@tom_elliff are you trying to install all your taps and targets into the same venv? Singer's recommended practice is to install each in its own, dedicated venv--for the problem you mention of conflicting subdependency versions.
Meltano will handle this best practice for you if you first get meltano installed and then do you tap and target installations using meltano to install them
t
this is just running the command 
meltano install
n
What is the result of 
which python
t
Copy code
$ which python
/home/tom/git/meltano-projects/.venv/bin/python
n
And 
pip freeze
?
And also which taps and targets have you enabled?
t
Copy code
$ pip freeze
aenum==2.1.2
aiohttp==3.4.4
alembic==1.0.11
aniso8601==8.0.0
argh==0.26.2
asn1crypto==1.3.0
async-generator==1.10
async-timeout==3.0.1
atomicwrites==1.2.1
attrs==20.2.0
Authlib==0.10
azure-common==1.1.25
azure-nspkg==2.0.0
azure-storage==0.36.0
Babel==2.8.0
backcall==0.2.0
backoff==1.8.0
bcrypt==3.1.6
blinker==1.4
boto3==1.8.9
botocore==1.11.9
Cerberus==1.2
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
Click==7.0
click-default-group==1.2.1
colorama==0.3.9
cryptography==3.1.1
decorator==4.4.2
docutils==0.16
fasteners==0.15
Flask==1.1.2
Flask-BabelEx==0.9.4
Flask-Cors==3.0.7
Flask-Executor==0.9.2
Flask-Login==0.5.0
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-RESTful==0.3.7
Flask-SQLAlchemy==2.3.2
Flask-WTF==0.14.3
flatten-dict==0.1.0
future==0.18.2
gunicorn==19.9.0
idna==2.7
ijson==3.1.2
ipython==7.5.0
ipython-genutils==0.2.0
itsdangerous==1.1.0
jedi==0.17.2
Jinja2==2.11.2
jmespath==0.10.0
jsonschema==2.6.0
Mako==1.1.3
Markdown==3.0.1
MarkupSafe==1.1.1
meltano==1.52.0
meltano-flask-security==0.1.0
monotonic==1.5
multidict==4.7.6
networkx==2.2
parso==0.7.1
passlib==1.7.2
pathlib2==2.3.5
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==2.0.10
psutil==5.6.3
psycopg2-binary==2.8.5
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodomex==3.9.8
Pygments==2.7.1
pyhocon==0.3.51
pyhumps==1.2.2
PyJWT==1.7.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
PyPika==0.25.1
python-dateutil==2.8.1
python-dotenv==0.14.0
python-editor==1.0.4
python-gitlab==1.8.0
pytz==2020.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.1.13
simplejson==3.16.0
six==1.15.0
smtpapi==0.4.1
snowflake-connector-python==1.6.10
snowflake-sqlalchemy==1.1.2
speaklater==1.3
SQLAlchemy==1.2.19
sqlparse==0.3.0
traitlets==5.0.4
typing-extensions==3.7.4.3
urllib3==1.23
watchdog==0.9.0
wcwidth==0.2.5
Werkzeug==0.16.1
WTForms==2.2.1
yarl==1.6.0
Copy code
$ grep name meltano.yml 
  - name: tap-gitlab
  - name: target-jsonl
  - name: target-postgres
d
@tom_elliff Can you delete your 
/home/tom/git/meltano-projects/.venv
venv, recreate it, and share the 
pip install meltano
output?
@tom_elliff Are you only seeing this error with 
meltano install
, or with other commands as well?
t
Untitled
Untitled
I also see it with 
meltano upgrade
but not 
meltano --version
or a bare 
meltano
I think this is probably a pyenv thing more than a meltano thing as there's a few threads about it online, just need to find something that works
d
@tom_elliff Yeah, it sounds like pyenv installed Python without libsqlite3. Did you see potential solutions https://stackoverflow.com/a/62270095 and https://forum.djangoproject.com/t/modulenotfounderror-no-module-named-sqlite3/952/6 as well?
Note that Meltano doesn't actually depend on SQLite if you use a postgres system database (https://meltano.com/docs/settings.html#database-uri), so that may be another option if fixing SQLite is not going to be worth the effort.
13 Views
Open in Slack
PreviousNext
Powered by