Are there any run stats emitted by meltano that I could use to determine if downstream processes should run in dagster? Is there a way to programatically distinguish between a successful
meltano run
that emitted 0 rows vs one that emitted 100?
I would only like to trigger a downstream dbt run and PowerBI refresh if there is anything new to process.
e
edgar_ramirez_mondragon
11/13/2023, 8:19 PM
Nothing explicit or as nice as a row count, but if you've only got incremental streams, you could try using the state as a proxy: if the state didn't change between runs, you could assume that nothing was updated. But do log a feature request or give https://github.com/meltano/meltano/issues/7085 (the closest thing imo) a 👍🏼.