https://linen.dev logo
Join Slack
Powered by
QQ: we’re adding a number of custom DAGs to the `o...
# best-practices
j
QQ: we’re adding a number of custom DAGs to the 
orchestrate/dags
directory that have package dependencies that aren’t shipped with Meltano/Airflow by default. I’ve added them using poetry (they’re tracked in poetry.lock and pyproject.toml), but they aren’t available in the custom DAGs. Am I missing something? I can’t really find any documentation for adding new dependencies either. 🤔
Error in Airflow:
Copy code
Broken DAG: [/project/orchestrate/dags/ml_iris_train_model.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/project/orchestrate/dags/ml_iris_train_model.py", line 4, in <module>
    from sklearn.multiclass import OneVsRestClassifier
ModuleNotFoundError: No module named 'sklearn'
Unless I’m thinking about this the wrong way and Meltano is not supposed to manage the dependencies for Airflow
e
For those dependencies to be in the same venv as airflow I think you'd have to override the 
pip_url
to include a 
requirements.txt
. Currently it's
Copy code
'apache-airflow==2.1.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt>'
So maybe something like
Copy code
'apache-airflow==2.1.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt> -r requirements.txt'
At this point you won't be able to install dependencies from poetry directly, but you can export 
poetry.lock
into a `requirements.txt`: https://python-poetry.org/docs/cli/#export
j
Sweet, that’s exactly what I was looking for! Thanks @edgar_ramirez_mondragon!
e
np @jobert_abma 😄
Open in Slack
PreviousNext
Powered by