I need some help with airflow Airflow is installed and the s Meltano #troubleshooting

I need some help with airflow. Airflow is installe...

mert_bakir

04/26/2023, 5:25 PM

I need some help with airflow. Airflow is installed and the scheduler is running

meltano invoke airflow scheduler -D

, then I created a job and set up daily schedule.

Copy code

schedules:
- name: test-schedule-01
  interval: '@daily'
  job: test-postgres-to-rds

How and when this will run? I don't see any dag on airflow ui. No dags folders exists. Do I need to create the dag myself, if so what does meltano schedules command do? I'm missing something here.

Copy code

orchestrate/
└── airflow
    ├── airflow-scheduler.err
    ├── airflow-scheduler.log
    ├── airflow-scheduler.out
    ├── airflow-scheduler.pid
    ├── airflow-webserver.err
    ├── airflow-webserver.log
    ├── airflow-webserver.out
    ├── airflow.cfg
    └── webserver_config.py

mert_bakir

04/26/2023, 5:31 PM

For more contenxt:

Copy code

Python 3.9.16
Meltano 2.18.0

Copy code

utilities:
  - name: airflow
    variant: apache
    pip_url: git+<https://github.com/meltano/airflow-ext.git@main> apache-airflow==2.3.3
      --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt>
    config:
      webserver:
        web_server_port: '8080'
      core:
        load_examples: 'False'

user

04/26/2023, 5:35 PM

Hey @mert_bakir 👋 . Did you follow the getting started section for airflow on the hub? It should have told you to run

meltano invoke airflow:initialize

which creates a dag generator in your project to automatically populate airflow. Do you have a dags folder in your

orchestrate

directory?

mert_bakir

04/26/2023, 5:38 PM

Hey @pat_nadolny, it looks like I was following the old link: https://hub.meltano.com/orchestrators/airflow/ . Let me set it up again.

user

04/26/2023, 5:41 PM

Ah yeah we recently transitioned to recommending utility plugins over orchestrators. If you saw any documentation that still isnt updated, would you mind opening an issue so we can get it fixed?

mert_bakir

04/26/2023, 5:48 PM

Good, it created the dags now. Thanks.

mert_bakir

04/26/2023, 5:49 PM

no problem, I'll open an issue for the old docs I see

Open in Slack

Previous Next