https://linen.dev logo
Join Slack
Powered by
I'm getting errors similar to what was encountered...
# troubleshooting
c
I'm getting errors similar to what was encountered by Gunnar in this thread , however, there are a few substantive differences.
I'm running in Docker and using PostgreSQL as my meltano/airflow database. Also, I expressly deleted and recreated the entire databse prior to deploying. Here's my 
orchestrator
config in `meltano.yml`:
Copy code
orchestrators:
  - name: airflow
    pip_url: psycopg2 apache-airflow==1.10.14 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt>
Here's the error I'm getting. Same odd reference to Airflow 2.0 and deprecated 
initdb
call (I actually tried using Airflow 2.x and it ironically did not flag the 
initdb
call): ```Installing extractor 'tap-postgres--telemetry'... [2021-06-25 054615,297] [7|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-postgres--telemetry' [2021-06-25 054615,298] [7|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-postgres--telemetry' ... VIRTUAL_ENV': '/project/.meltano/orchestrators/airflow/venv', 'AIRFLOW_HOME': '/project/.meltano/run/airflow', 'AIRFLOW_CONFIG': '/project/.meltano/run/airflow/airflow.cfg'} [2021-06-25 054646,008] [878|MainThread|root] [DEBUG] Deleted configuration at /project/.meltano/run/airflow/airflow.cfg [2021-06-25 054646,009] [878|MainThread|meltano.cli.invoke] [ERROR] The 'initdb' command is deprecated and removed in Airflow 2.0, please use 'db init' instead INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. Traceback (most recent call last): File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/base.py", line 162, in _catch_revision_errors yield File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/base.py", line 364, in _upgrade_revs revs = list(revs) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 765, in _iterate_revisions requested_lowers = self.get_revisions(lower) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 319, in get_revisions return sum([self.get_revisions(id_elem) for id_elem in id_], ()) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 319, in <listcomp> return sum([self.get_revisions(id_elem) for id_elem in id_], ()) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 324, in get_revisions for rev_id in resolved_id File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 324, in <genexpr> for rev_id in resolved_id File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 397, in _revision_for_ident resolved_id, alembic.script.revision.ResolutionError: No such revision or branch 'd135f52a6f49' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/project/.meltano/orchestrators/airflow/venv/bin/airflow", line 37, in <module> args.func(args) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/cli.py", line 233, in wrapper func(args) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/bin/cli.py", line 1626, in initdb db.initdb(settings.RBAC) File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/db.py", line 323, in initdb upgradedb() File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/db.py", line 386, in upgradedb command.upgrade(config, 'heads') File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade script.run_env() File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env util.load_python_file(self.dir, "env.py") File "/project/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file module = load_module_py(module_id, path) File "/project/.meltano/orchestrators…
I tried running 
meltano invoke airflow db upgrade
after 
meltano install
and before calling the Airflow scheduler, but that didn't work (though it was when I was trying to use the 
2.1.0
version.
I tried obtaining the airflow version at runtime via 
meltano invoke airflow vesion
, but it produced nothing in the way of output, only env vars. Speaking of, here is a subset that might prove helpful:
Copy code
[2021-06-25 06:27:33,070] [874|MainThread|root] [DEBUG] 
Env: {
    'PATH': '/project/.meltano/orchestrators/airflow/venv/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 
    'PYTHON_VERSION': '3.7.10', 'PYTHON_PIP_VERSION': '21.1.2', 
    'NODE_VERSION': '10', 'DEBIAN_FRONTEND': 'noninteractive', 'PYTHONDONTWRITEBYTECODE': '1', 'PYTHONUNBUFFERED': '1', 'MELTANO_PROJECT_READONLY': 'true', 'HOME': '/root', 
    'AIRFLOW__CORE__EXECUTOR': 'LocalExecutor', 
    'AIRFLOW__CORE__SQL_ALCHEMY_CONN': 'postgresql://<redacted>', 'MELTANO_DATABASE_URI': 'postgresql://<redacted>', 
    'MELTANO_UI_SERVER_NAME': '<redacted>', 'MELTANO_UI_SESSION_COOKIE_SECURE': 'true', 
    'MELTANO_PROJECT_ROOT': '/project', 'MELTANO_ORCHESTRATOR_NAME': 'airflow', 'MELTANO_ORCHESTRATOR_NAMESPACE': 'airflow', 'MELTANO_ORCHESTRATOR_VARIANT': 'original', 'AIRFLOW__CORE__DAGS_FOLDER': '/project/orchestrate/dags', 'AIRFLOW_CORE_DAGS_FOLDER': '/project/orchestrate/dags',        
    'MELTANO_ORCHESTRAT_CORE_DAGS_FOLDER': '/project/orchestrate/dags', 'AIRFLOW__CORE__PLUGINS_FOLDER': '/project/orchestrate/plugins', 'AIRFLOW_CORE_PLUGINS_FOLDER': '/project/orchestrate/plugins', 'MELTANO_ORCHESTRAT_CORE_PLUGINS_FOLDER': '/project/orchestrate/plugins', 
    'AIRFLOW_CORE_SQL_ALCHEMY_CONN': 'postgresql://<redacted>', 'MELTANO_ORCHESTRAT_CORE_SQL_ALCHEMY_CONN': 'postgresql://<redacted>', 'AIRFLOW__CORE__LOAD_EXAMPLES': 'False', 'AIRFLOW_CORE_LOAD_EXAMPLES': 'False', 'MELTANO_ORCHESTRAT_CORE_LOAD_EXAMPLES': 'False', 
    'AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION': 'False', 'AIRFLOW_CORE_DAGS_ARE_PAUSED_AT_CREATION': 'False', 'MELTANO_ORCHESTRAT_CORE_DAGS_ARE_PAUSED_AT_CREATION': 'False', 
    'VIRTUAL_ENV': '/project/.meltano/orchestrators/airflow/venv', 'AIRFLOW_HOME': '/project/.meltano/run/airflow', 'AIRFLOW_CONFIG': '/project/.meltano/run/airflow/airflow.cfg'
}
[2021-06-25 06:27:34,872] [874|MainThread|root] [DEBUG] Deleted configuration at /project/.meltano/run/airflow/airflow.cfg
[2021-06-25 06:27:34,873] [874|MainThread|meltano.cli.invoke] [ERROR] The 'initdb' command is deprecated and removed in Airflow 2.0, please use 'db init' instead
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
It looks like, from local debugging, that whitespace is introduced into the version string and so it's flagged as invalid by 
StrictVersion
Copy code
Traceback (most recent call last):
  File "/home/chb/.pyenv/versions/vanilla-meltano/lib/python3.7/site-packages/meltano/cli/__init__.py", line 52, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: invalid version number '1.10.14'
message has been deleted
a
Thanks for reporting this @casey. I'm taking a look... Also, in case it turns out to be related, I'm linking to this existing issue where 'StrictVersion' is raised although it was not actually the root cause. `meltano invoke airflow` fails with `StrictVersion` error when Airflow is misconfigured (#2758) · Issues · Meltano / Meltano · GitLab
c
Thanks, @aaronsteers. I'm trying to find where in the codebase the 
pip_url
is parsed (that seems to be the way that Meltano is informed of the airflow version). The closest I've come is 
venv_service.py
, but I don't think the special case of the airflow plugin is treated there.
d
2 Views
Open in Slack
PreviousNext
Powered by