Hi all! I am currently using Airflow as our orchestrator. And I'm trying to set up some basic email alerting for our Airflow DAG runs (ideally on success and failure. But looking for just on failed DAG runs to start)! I have found some guides online that explain I can edit the

DEAULT_ARGS

to look as follows....

DEFAULT_ARGS = {
    "owner": "airflow",
    "depends_on_past": False,
    "email": ['<mailto:connor@gmail.com|connor@gmail.com>'],
    "email_on_failure": True,
    "email_on_retry": False,
    "catchup": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
    "concurrency": 1,
}

I have tried to make these changes in the

meltano.py

file under the path

meltano-project/*orchestrate/dags/*

(As the code that automatically defines our meltano DAGs is in the file

meltano.py

not

airflow.cfg

) . However after making these changes to the

meltano.py

file, invoking meltano scheduler and manually trigger a DAG (which failed) I did not receive any emails to the address I defined (

<mailto:connor@gmail.com|connor@gmail.com>

) ! If anyone has any guides or solutions to Airflow email alerting with meltano projects that would be greatly appricated!

Hey @connor_flynn ! - You can configure airflow using environment variables or settings configs then I think Meltano manages compiling, writing, and making the config accessible to airflow for you - see https://docs.meltano.com/guide/production#airflow-orchestrator. Its been a while since I've used email alerting but you do need configure your smtp server properly for it to work https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#smtp and that its set to send on failure https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#email

Thank you for the prior tips @pat_nadolny! re: the missing

airflow.cfg

... We are having issues with our Airflow scheduler. When running

meltano invoke airflow scheduler -D

(in order to start the airflow scheduler in the background). Despite producing no errors (seen in screenshot) the Airflow Ui shows the "scheduler has no heartbeat". I am trying to figure out how meltano creates the

airflow.cfg

and how it would be removed with no pattern and how file this could effect the scheduler issues...If you have any suggestion that would be greatly appreciated!