Hey community! I'm deploying meltano using airflow...
# announcementsm
murilo_tavolaro_de_nigris
10/07/2020, 1:21 PMHey community! I'm deploying meltano using airflow as orchestrator and postgres as DB into production using the docker-compose file.
However when I try to UP all services, I get the following error:
As Meltano was the one who installed airflow and all its dependencies in .meltano, I don't where/how to add the psycopg2 library
n
niall_woodward
10/07/2020, 1:50 PMHey you need to add this alongside airflow in the meltano.yml file:
Copy code
orchestrators:
- name: airflow
pip_url: psycopg2==2.8.6 apache-airflow==1.10.12niall_woodward
10/07/2020, 1:51 PMMake sure to keep the airflow version fixed at what it is currently, not necessarily the version in my comment. Also make sure you delete the airflow venv in .meltano/orchestrators/airflow
m
murilo_tavolaro_de_nigris
10/07/2020, 2:42 PMThank you @niall_woodward!!
d
douwe_maan
10/07/2020, 3:49 PM@niall_woodward Is this always necessary when using Airflow with a Postgres metadata DB? If so, we should document that, but I was assuming it'd just work OOTB
n
niall_woodward
10/07/2020, 3:54 PMThat's just what I've been doing 😄 but after a bit more investigation there is a postgres package you're meant to install! https://airflow.apache.org/docs/stable/installation.html#extra-packages
niall_woodward
10/07/2020, 3:54 PMSo @murilo_tavolaro_de_nigris see how you get on without pyscopg2 and adding the postgres package to your airflow pip_url? So apache-airflow[postgres]==x
niall_woodward
10/07/2020, 3:59 PMIgnore all the above you need the pyscopg2 not the postgres package, that's just for airflow operators.
niall_woodward
10/07/2020, 4:02 PMYou also need database client packages (Postgres or MySQL) if you want to use those databases.
d
douwe_maan
10/07/2020, 4:06 PM@niall_woodward Good catch! We should mention that under https://meltano.com/docs/production.html#airflow-orchestrator, then: https://gitlab.com/meltano/meltano/-/issues/2378. I'll add it today.
m
murilo_tavolaro_de_nigris
10/07/2020, 4:50 PMThank you!! It worked once I added psycopg2 🙂
8 Views