https://linen.dev logo
Join Slack
Powered by
Airflow + Meltano problem After running `meltano ...
# troubleshooting
d
Airflow + Meltano problem After running 
meltano invoke airflow scheduler
I have this error 
ImportError: cannot import name 'Mapped' from 'sqlalchemy.orm' (/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/__init__.py)
Anyone had something similar? I will add logs in discussion
```╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ │ │ /usr/local/bin/meltano:5 in <module> │ │ │ │ 2 # -- coding: utf-8 -- │ │ 3 import re │ │ 4 import sys │ │ ❱ 5 from meltano.cli import main │ │ 6 if name == '__main__': │ │ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │ │ 8 │ sys.exit(main()) │ │ /usr/local/lib/python3.9/site-packages/meltano/cli/__init__.py:10 in <module> │ │ │ │ 7 import sys │ │ 8 import typing as t │ │ 9 │ │ ❱ 10 from meltano.cli.utils import CliError │ │ 11 from meltano.core.error import MeltanoError, ProjectReadonly │ │ 12 from meltano.core.logging import setup_logging │ │ 13 │ │ │ │ /usr/local/lib/python3.9/site-packages/meltano/cli/utils.py:16 in <module> │ │ │ │ 13 from click_default_group import DefaultGroup │ │ 14 │ │ 15 from meltano.core.error import MeltanoConfigurationError │ │ ❱ 16 from meltano.core.logging import setup_logging │ │ 17 from meltano.core.plugin import PluginType │ │ 18 from meltano.core.plugin.error import PluginNotFoundError │ │ 19 from meltano.core.plugin_install_service import ( │ │ │ │ /usr/local/lib/python3.9/site-packages/meltano/core/logging/__init__.py:5 in <module> │ │ │ │ 2 from future import annotations │ │ 3 │ │ 4 from .formatters import console_log_formatter, json_formatter, key_value_formatter │ │ ❱ 5 from .job_logging_service import ( …
s
sounds like an sqlalchemy version problem. Have you tried python 3.8 (which automatically hopefully resolves to a different sqlalchemy version as we're running it in production).
d
Tried to reinstall it had another error, python world is a hell of a mess ```‹main*› »»»» meltano invoke airflow scheduler 0|165604 2023-03-29T145608.975766Z [info ] Environment 'dev' is active 2023-03-29T145613.837503Z [error ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. Command 
airflow version
failed [I] [~/M/b/data-pipeline|system|3.8.16] ⚡️ 9,245s ‹main*› »»»» airflow version 1|165615 /usr/local/lib/python3.9/site-packages/airflow/configuration.py:441 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:468 FutureWarning: The 'dag_default_view' setting in [webserver] has the old default value of 'tree'. This value has been changed to 'grid' in the running config, but please update your config before Apache Airflow 3.0. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:347 DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:468 FutureWarning: The 'auth_backends' setting in [api] has the old default value of 'airflow.api.auth.backend.deny_all'. This value has been changed to 'airflow.api.auth.backend.session' in the running config, but please update your config before Apache Airflow 3.0. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:468 FutureWarning: The 'log_id_template' setting in [elasticsearch] has the old default value of '{dag_id}-{task_id}-{execution_date}-{try_number}'. This value has been changed to '{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}' in the running config, but please update your config before Apache Airflow 3.0. /usr/local/lib/python3.9/site-packages/airflow/configuration.py:361 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/settings.py:250 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/settings.py:338 DeprecationWarning: The sql_alchemy_pool_enabled option in [core] has been moved to the sql_alchemy_pool_enabled option in [database] - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/settings.py:401 DeprecationWarning: The sql_engine_encoding option in [core] has been moved to the sql_engine_encoding option in [database] - the old setting has been used, but please update your config. /usr/local/lib/python3.9/site-packages/airflow/cli/cli_parser.py:905 DeprecationWarning: The namespace option in [kubernetes] has been moved to the namespace option in [kubernetes_executor] - the old setting has been used, but please update your config. 2.5.2 [I] [~/M/b/data-pipeline|system|3.8.16] ⚡️ 0,742s ‹main*› »…
@Sven Balnojan I resolved the issue by using python 3.8.16 but also I have a feeling that I saw another one I had installed airflow separately as a python package (the program, not the lib) with this option I was not able to run Airflow, (As I understand with his option meltano will call the binary in system ?)
• use the `LocalExecutor` instead of the 
SequentialExecutor
default by setting the 
core.executor
setting (or 
AIRFLOW__CORE__EXECUTOR
environment variable) to `LocalExecutor`:
• meltano config airflow set core.executor LocalExecutor
•
• export AIRFLOW__CORE__EXECUTOR=LocalExecutor
•
meltano showed an error that it couldn't find the binary
Copy code
‹main*› »»»» meltano invoke airflow scheduler                                                                                   0|16:56:04
2023-03-29T14:56:08.975766Z [info     ] Environment 'dev' is active
2023-03-29T14:56:13.837503Z [error    ]
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Command `airflow version` failed
But when I removed 
LocalExecutor
setting, I was able to run airflow
s
@dima_petukhov it's best to completely separate your meltano project from "your system". I always recommend to dockerize it completely, but at least use venvs (we recommend using pipx), Python dependency management isn't pretty, and when we're talking about moving data, well, then there is a whole hell of dependencies to take account for. So best to completely isolate them to at least be certain what is installed & used.
d
Thank you Sven, I agree with you. Btw, I've resolved the installation problem via pipx
Open in Slack
PreviousNext
Powered by