Hi all, I am trying to build a Docker image of my project, which includes Airflow. I am seeing this message:

Orchestrator 'airflow' could not be installed: failed to install plugin 'airflow'

Which is then followed by a section about psycopg2 and pg_config.

#8 32.06   error: subprocess-exited-with-error
#8 32.06   
#8 32.06   × python setup.py egg_info did not run successfully.
#8 32.06   │ exit code: 1
#8 32.06   ╰─> [23 lines of output]
#8 32.06       running egg_info
#8 32.06       creating /tmp/pip-pip-egg-info-25bc6asr/psycopg2.egg-info
#8 32.06       writing /tmp/pip-pip-egg-info-25bc6asr/psycopg2.egg-info/PKG-INFO
#8 32.06       writing dependency_links to /tmp/pip-pip-egg-info-25bc6asr/psycopg2.egg-info/dependency_links.txt
#8 32.06       writing top-level names to /tmp/pip-pip-egg-info-25bc6asr/psycopg2.egg-info/top_level.txt
#8 32.06       writing manifest file '/tmp/pip-pip-egg-info-25bc6asr/psycopg2.egg-info/SOURCES.txt'
#8 32.06       
#8 32.06       Error: pg_config executable not found.
#8 32.06       
#8 32.06       pg_config is required to build psycopg2 from source.  Please add the directory
#8 32.06       containing pg_config to the $PATH or specify the full executable path with the
#8 32.06       option:
#8 32.06       
#8 32.06           python setup.py build_ext --pg-config /path/to/pg_config build ...
#8 32.06       
#8 32.06       or with the pg_config option in 'setup.cfg'.
#8 32.06       
#8 32.06       If you prefer to avoid building psycopg2 from source, please install the PyPI
#8 32.06       'psycopg2-binary' package instead.
#8 32.06       
#8 32.06       For further information please check the 'doc/src/install.rst' file (also at
#8 32.06       <<https://www.psycopg.org/docs/install.html>>).
#8 32.06       
#8 32.06       [end of output]
#8 32.06   
#8 32.06   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 32.06 error: metadata-generation-failed
#8 32.06 
#8 32.06 × Encountered error while generating package metadata.
#8 32.06 ╰─> See above for output.
#8 32.06 
#8 32.06 note: This is an issue with the package mentioned above, not pip.
#8 32.06 hint: See above for details.
#8 32.06

I already have the

psycopg2-binary

installed.

(venv) meltano-project1$ pip install psycopg2-binary
Requirement already satisfied: psycopg2-binary in ...

How do I get Airflow to install correctly?

pipx install "apache-airflow==2.5.2" --python $(which python)

Thank you. I'll give that a try.

@ian_lewis I've changed script, it's important to pass python version needed for airflow it's a whole mess with python versions

Ah, OK. Thank you! I'm also trying to work out why when I try to use the Meltano docker-compose my Airflow webserver and scheduler keep crashing and restarting :/

@ian_lewis could you share your logs?

The important bit is...

Executable 'airflow' could not be found. Orchestrator 'airflow' may not have been installed yet using 'meltano install orchestrator airflow,' or the executable name may be incorrect

@ian_lewis I did not do this installation, did you check the official docker-compose from airflow just for inspiration ?

I am learning that! 🙂

Hi @ian_lewis, Were you able to solve this? I am facing same issue with docker compose. Please let me know

Hi @pandyarajan_vv I have decided that it is better in terms of reducing complexity and avoiding spaghetti config to keep Meltano (and dbt) separate when using Airflow. I'm sure there is (or will be) an easy approach that allows Airflow to act as the meltano context but the issues when handling dependencies and the use of old releases make things difficult when trying to deal with security patches etc.