Is there any way to run a command, or some other hook, whenever an ELT pipeline completes?

We don't have hooks nativelly supported by Meltano right now, but we are using Airflow for running the pipelines, so maybe there is a way to add a hook there.

ah does the manual "run once" button actually trigger airflow behind the scenes?

That's a great question --> I'll check that tomorrow morning when I get back to my meltano laptop. @derek_knox any quick insights on this one? Do we start a proper pipeline on

run once

from Meltano UI?

I'll look in the source code for where I can add a hook on ETL completion

@luke_marsden The data pipeline (schedule) you create through the UI or CLI is an abstraction we use so Meltano can technically use any Orchestrator. Currently, we're using Airflow as that Orchestrator. As I recall (this is @micael_bergeron wheelhouse) Airflow crawls a set of directories and automatically generates DAGs based on what it finds in its crawl. Since Airflow can read our data pipeline (schedule) abstraction, it can auto create a DAG using its key-value pairs.

@once

is one of them (https://airflow.apache.org/scheduler.html#dag-runs). As of last week https://gitlab.com/meltano/meltano/merge_requests/1138 Airflow should be triggered automatically behind the scenes.

ah, cool!

Thinking... out of curiosity what are you trying to trigger? This idea of triggers came up, we obviously just haven't pursued it yet.

I'm interested in snapshotting the database each time a pipeline runs

When you go through the UI and get feedback in the Job Log modal. You can see that once it completes (ELT has succeeded) we use that hook. We are polling the job status and that's our current implementation of knowing when a specific ELT job is complete.

ok, which hook specifically? do you mean you are polling airflow for status?

ooh typed Python, nice

Yes, when an ELT run gets kicked off it creates a job entry in the application db. Then we poll via this endpoint until we get

is_complete

confirmation.

ok, when you say "we" do you mean in the UI, or somewhere else?

Please submit a feature request! Again @micael_bergeron is most knowledgable here and it may be a simple addition. At the very least we can get it queued (as I stated this idea has bumped around, just hasn't been prioritized)

cool - thanks - will do!

No worries on the architectural understanding, that's non-trivial code bases for ya

cool

Just have had more exposure to it naturally

Vue.js is nice!

Agreed, Vuex is ace too (MobX is too if you haven't heard of it, but not in our codebase as Vuex does the job atm)

I'll check it out 🙂

ah does the manual "run once" button actually trigger airflow behind the scenes?

No it doesn't, it is only a ELT one

I wonder what the cleanest way to configure meltano or airflow to run an extra command after ELT completion...

You can actually extend the Meltano DAG located in

orchestrate/dags/meltano.py

. I haven't tried out myself, but I wonder if you could maybe extend the DAG definition or add a dependency in a different file.