Has anyone successfully been able to install the latest version of airflow

2.1.0

with meltano? I'm getting this dependency issue.

I was able to install it using an appropriate constraints file reference in the

pip_url

, but I couldn't get it to run properly.

pip_url: psycopg2 apache-airflow==2.1.0 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.0/constraints-3.7.txt>

Are you using a Mac M1? If so I'd recommend using the docker image as it is stable!

@lars It looks like you may be trying to install apache-airflow and meltano into the same virtual environment, instead of installing just Meltano, and then installing Airflow through Meltano, so that it gets its own dedicated environment. If you add Airflow as a plugin to Meltano and use the

pip_url

Casey has shared, you shouldn’t see any dependency incompatibility issues

Thanks for the info @casey I'm also running the Postgers related tap so my set-up looks very similar. @douwe_maan I also installed the

apache-airflow-providers-slack==2.0.0

into the airflow environment to have slack notifications for failures so that was the culprit of why I had dependency issues.

@lars Ah, OK. I think adding that to

pip_url

for

airflow

in

meltano.yml

instead should make sure it only makes it into the Airflow env, and not Meltano’s own

Hello @lars @douwe_maan, how did you install

apache-airflow-providers-slack

, I installed via this but it's not working.

- name: airflow
    pip_url: apache-airflow-providers-slack==2.0.0 psycopg2 apache-airflow==2.1.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt>
    config:
      webserver:
        authenticate: true
        rbac: true
        auth_backend: airflow.contrib.auth.backends.password_auth

@monika_rajput are you seeing an error message when you run “meltano install”?

@douwe_maan Yes i am facing version conflicts error.

@monika_rajput All right, it looks like a legitimate incompatibility between the versions of airflow and the slack provider; not something specific to Meltano. Can you find different versions that are documented as being compatible?

Yes i checked the version compatibility PIP package Version required apache-airflow >=2.1.0 slack_sdk >=3.0.0 https://pypi.org/project/apache-airflow-providers-slack/

In the Meltano.yml snippet you shared earlier, the airflow version was set to 2.1.2, but that should still be compatible. Are you seeing the same error when you run “pip install <that pip_url>” outside of Meltano?

When i was using pip install

pip install apache-airflow-providers-slack==4.2.3 psycopg2 apache-airflow==2.1.2,

it install the package successfully but in airflow UI giving

No module named 'jinja2.debug'

Ah good to know, does that version of the slack provider install properly in meltano as well?

Looking at the contents of the constraints file https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.9.txt,

apache-airflow-providers-slack==4.0.0

is in there and the message I'm seeing is

ERROR: Cannot install apache-airflow-providers-slack==2.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested apache-airflow-providers-slack==2.0.0
    The user requested (constraint) apache-airflow-providers-slack==4.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

@douwe_maan @edgar_ramirez_mondragon, Thank you so much i was able to install the slack provider but strange now i am getting

from airflow.operators.bash_operator import BashOperator

ModuleNotFoundError: No module named 'airflow.operators'

@monika_rajput That suggests that your project’s orchestrate/dags/meltano.py is incompatible with the version of airflow you're using. You can upgrade it with “meltano upgrade --files” if I'm not mistaken

It resolved by removing

*from* airflow.operators.bash_operator *import* BashOperator

and add

*from* airflow.operators.bash *import* BashOperator

instead. Thank you so much. @douwe_maan , and thank you @edgar_ramirez_mondragon for sharing the version file that really helps me a lot.🙏