Welcome to the channel to share what you got! I've just stumbled over a Meltano - Prefect workflow created by @Henning Holgersen, and I really wanted a place for people to share these kinds of things. More people should these this stuff. (1) You can find @Henning Holgersen’s creation here: https://github.com/radbrt/prefect_flows/tree/main/projects/jottings/meltano_flow

Happy that this is useful, we are in a way going in a different direction than many with Meltano, trying to do all scheduling in prefect and avoid running our own servers. One thing I want to mention, is that this workflow would benefit from the upcoming

--no-colors

flag that is in development. The logs are a little verbose, and includes a lot of misevaluated characters due to the coloring.

@Henning Holgersen working with some folks doing a very very similar to setup to you, same issue with no colors. There's atleast one other company doing this 😄

@visch Great to hear that we aren’t completely on our own. Would love to see how others are doing this. Are they going with Prefect? Airflow? Dagster?

Prefect

We do this with Airflow

I never actually got around to trying Airflow, one of my colleagues vetoed it early on when we were deciding 🤔 GCS=Google Cloud storage? We are (for now at least) using a small stateful k8s postgres DB.

fwiw, if this workflow works for you, thats great! Meltanos EL features are one of the post powerful parts, and it is easy to dockerize these. On the deployment side we do currently fall short (on documentation ;)), and if you already got airflow/dagster/prefect running, this seems like a good option. I'm tempted to even include this into the docs.

Maybe we can create a blog post or something when the --no-color option is launched? And I’d be happy to doodle a little more around deployment patterns (which is tool-specific). The example I have builds the entire meltano flow into the image, but there might be cleaner ways of doing this.