Hi all I m having issues with running meltano with Airflow F Meltano #getting-started

Hi all, I'm having issues with running meltano wit...

mathieu_hinderyckx

08/25/2022, 11:39 AM

Hi all, I'm having issues with running meltano with Airflow. Followed most of the getting started guides and docs, then moved the

system database

to a postgres instance, and I can use that with

meltano run ...

, so far so good. Now I want to add the airflow executor. I've followed the docs and set

AIRFLOW__CORE__SQL_ALCHEMY_CONN

to the same string as the system database. However when running anything with Airflow, I now get an error consistently along the lines of ```The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/bin/airflow", line 8, in <module> sys.exit(main()) File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/__main__.py", line 40, in main args.func(args) File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 48, in command return func(*args, **kwargs) File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb db.initdb() File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/utils/db.py", line 563, in initdb upgradedb() File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/utils/db.py", line 702, in upgradedb command.upgrade(config, 'heads') File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/command.py", line 294, in upgrade script.run_env() File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/script/base.py", line 490, in run_env util.load_python_file(self.dir, "env.py") File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file module = load_module_py(module_id, path) File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/util/compat.py", line 184, in load_module_py spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/migrations/env.py", line 115, in <module> run_migrations_online() File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/migrations/env.py", line 106, in run_migrations_online context.run_migrations() File "<string>", line 8, in run_migrations File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/runtime/environment.py", line 813, in run_migrations self.get_context().run_migrations(**kw) File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/runtime/migration.py", line 549, in run_migrations for step in self._migrations_fn(heads, self): File "/Users/mathieu/repositories/otainsight/meltano/my-meltano-project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/alembic/comm…

mathieu_hinderyckx

08/25/2022, 12:33 PM

It seems like meltano and airflow cannot use the same database? If I use the same connection string for the database_uri and for the airflow connection, then I get this error and it breaks.

TomasB

08/25/2022, 1:00 PM

You need another DB for the airflow

TomasB

08/25/2022, 1:01 PM

Also you might need to change from

AIRFLOW__CORE__SQL_ALCHEMY_CONN

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

mathieu_hinderyckx

08/25/2022, 2:15 PM

So we can't run both meltano UI and airflow on the same project?

mathieu_hinderyckx

08/25/2022, 2:16 PM

On your latter remark: neither

AIRFLOW__CORE__SQL_ALCHEMY_CONN

nor

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

work, I get this error:

Copy code

mathieu@OTA-Mathieu-MBP-7MD6R my-meltano-project % meltano invoke airflow webserver        
2022-08-25T14:16:12.961703Z [info     ] Environment 'dev' is active
2022-08-25T14:16:15.964793Z [error    ] 
Command `airflow version` failed

If I use the same connection string in my meltano.yml file under _`core.sql_alchemy_conn`,_ then it does work

TomasB

08/25/2022, 2:20 PM

You can run both meltano UI and airflow on the same project the DB uris though should be similar to :

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: <postgresql://postgres:postgres@airflow-metadata-db/airflow>

MELTANO_DATABASE_URI: <postgresql://postgres:postgres@meltano-system-db/meltano>

As you can see the meltano UI db is different from airflow db

mathieu_hinderyckx

08/25/2022, 2:21 PM

yeah but then they won't share the same state information about pipelines?

alexander_butler

08/27/2022, 9:09 PM

A single Postgres instance can house both databases which is of course more of a logical separation. What state info are you referring to specifically? They both use meltano job/schedule data. And because airflow is just running meltano cli commands, those commands are using meltano db

6 Views

Open in Slack

Previous Next