If I switch from using Airflow to another tool like Prefect.io to orchestrate my Meltano pipelines, will I lose the convenience of being able to see the pipeline details in the Meltano UI? For example, seeing things like the log output, last run date and so on

Yes, I believe Airflow is the only supported orchestrator for Meltano. We use Prefect and use their UI. I have plans to expose some of this information to my customers, so I'm interested if anything gets started along the lines of making Prefect a first-class citizen of Meltano

@al_whatmough The Pipelines UI doesn't depend on any particular orchestrator, it only requires that your schedules are defined in

meltano.yml

(using

meltano schedule

or the UI itself), and that you provide those schedule names to

meltano elt

as the

--job_id

. Storing logs, last run date, etc. is all the responsibility of

meltano elt

itself, and it doesn't matter what orchestrator is used to run the command. This means that you can get Meltano and the UI to work with any orchestrator, as long as you can configure it to iterate over the output of

meltano schedule list --format=json

and create a scheduled job of some kind for each entry, with shell command

meltano elt {schedule["elt_args"]}

. That's exactly what the default Airflow DAG generator for Meltano does, which is automatically added to your project at

orchestrate/dags/meltano.py

when you run `meltano add orchestrator airflow`: https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py The rest of Meltano's Airflow-specific code (https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py and https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L876-890) only serves to automatically build the

airflow.cfg

config file and tell Airflow to look for DAGs in the project's

orchestrate/dags

directory.

Thanks @adam_roderick and @douwe_maan! Sure, I can create an issue for this. Also, I was wondering if either of you can think of any reason why Prefect Cloud wouldn't work for this (as opposed to running Prefect myself in a virtualenv or Docker container)?

@al_whatmough we use prefect cloud for this

Would be awesome if we could have some documentation on using Prefect in the orchestration section of the Meltano docs. I haven't actually used Prefect yet - just been reading up on it. So I'm probably not the right person to write the docs 😅

Ideally, we'd support

meltano add orchestrator prefect

and use a file bundle to automatically add the appropriate file(s) to the project that would register a Prefect workflow for each schedule. I think the user would still need to configure Prefect to connect with the API or Cloud, and they'd need to explicitly run that file to register the workflows, which we'd document. Before we start documenting, though, let's see how much of this we can automate by using Meltano's configuration, file bundle, and

meltano invoke

bits 🙂