Hello channel, Building the docker image using a l...
# docker
p
Hello channel, Building the docker image using a local meltano project directory and using this image to launch airflow-webserver and airflow-scheduler fails with the following error (when you use postgresql database in AIRFLOW__CORE__SQL_ALCHEMY_CONN as given in docker-compose.prod.yml file generated by docker compose file bundle:
Copy code
[2021-02-09 22:25:17,544] [1|MainThread|meltano.cli.invoke] [ERROR] Traceback (most recent call last):
File "/project/.meltano/orchestrators/airflow/venv/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/airflow/settings.py", line 439, in initialize
configure_orm()
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/airflow/settings.py", line 250, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args)
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
return strategy.create(*args, **kwargs)
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/project/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 778, in dbapi
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
Airflow metadata database could not be initialized: `airflow initdb` failed
I tried adding
psycopg2-binary==2.8.6
in requirements.txt file, but the build log says
Requirement already satisfied: psycopg2-binary==2.8.6 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (2.8.6)
.
Using the default SQLite airflow db works okay, (removing the _`AIRFLOW__CORE__SQL_ALCHEMY_CONN` env and setting_
AIRFLOW__CORE__EXECUTOR: SequentialExecutor
in docker-compose.prod.yml) works okay, but i want to use an external postgresdb for this.
d
See https://meltano.com/docs/production.html#airflow-orchestrator, point 2:
For this to work, the `psycopg2` package will also need to be installed alongside `apache-airflow` , which you can realize by adding
psycopg2
to `airflow`'s
pip_url
in your `meltano.yml` project file (e.g.
pip_url: psycopg2 apache-airflow
) and running
meltano install orchestrator airflow
.
We should definitely make it more clear that that also applies when using the production docker-compose image, probably under here: https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/README.md#usage
Thanks for bringing this up, I've made notes to document this more clearly!
p
Amazing! Thanks Douwe.