Hello. When running Meltano + Airflow (confirmed the scheduler is active+running), clicking Run Now in Meltano UI on a Daily pipeline seems to run the pipeline using Meltano, as opposed to triggering the pipe’s Airflow DAG. Is that normal/expected?

Douwe will know more but I’m not entirely surprised that would be the case. We haven’t been focused on the UI so it’s possible something has diverged. Will you file an issue in https://gitlab.com/meltano/meltano/-/issues/ describing the behavior and what you’d expect?

@michael_schonfeld This is expected right now, since Meltano UI doesn't know if an orchestrator is running and will just start the job inside the web process. That's obviously not ideal 🙂 Can you please file an issue like Taylor suggested? I like the idea of triggering the orchestrator

You got it! Thanks for the quick responses, as always

@michael_schonfeld I've left a comment on the issue describing what the implementation could look like. If you'd like to have a go at that, that'd be greatly appreciated!

@douwe_maan what do you think is the best way to detect the presence or airflow orch — checking for the

airflow-scheduler.pid

file?

@michael_schonfeld Airflow may very well be running in a different Docker container with its own run dir, so that wouldn't work in a lot of cases. I think our best bet is to just assume that the Airflow scheduler is running when the

airflow

plugin is installed in the project, since that's a pretty explicit indication that Airflow will be used to handle the actual orchestration of pipelines

do you think this “always run in airflow” feature should be a bool configurable in the airflow plugin settings?

Hmm, what circumstances are you imagining where that wouldn't be the desired behavior?

fair point

@michael_schonfeld Thanks!