hey folks, does

meltano invoke airflow scheduler

pickup all the dags in the

orchestrate/dags

folder? looks like one of them isnt getting picked up. I have two in there

@kevin yes airflow will scan all the dags in your

orchestrate/dags

folder. Is it possible that the dag not loading has errors, have you checked the log? Also this airflow setting https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-dir-list-interval defines how frequently the directory is checked for new dags, how long has airflow been running? Is it possible that it hasnt had a chance to reload the file yet?

Hey @pat_nadolny yeah I did check the logs and it seems to be credentials relating to the extractor. My next post shows that which im currently trying to debug

@kevin have you run this pipeline outside of Airflow with success? Maybe try running it with

meltano --log-level=debug elt --dump=extractor-config ...

to see what the config looks like and if its missing anything

yeah the configs are correct and when i run meltano elt with that specific tap it works. Something about meltano airflow scheduler seems to not pickup one of them

Whats the reason for adding a custom dag? Is it possible to share your dag file? I wonder if theres any issues with the actual dag thats causing it to fail