Great article, thank you for sharing! I was deep into this ~2 years ago and somehow didn't hear about Kubeflow/Metaflow 😮, maybe they are newer than ~2 years? Haven't heard the category "Infrastructure abstraction" Kubeflow/Metaflow are super curious!

Yeah, this is a big one IMO. At the very least, we need to have better cookbooks for how to set up dev/prod environments for data scientists. Like managing cloud infrastruftue, getting an orchestration environment up and running was one of those surprisingly challenging tasks. We went with Argo, but I was investigating Dagster recently, and even with my familiarity with the space, it looks to me like a massive endeavor to get working (securely) -- have to build a way to containerize your environment, push to a repo, manage secrets, increment versions... it's a ton of work

cc @ken_payne @aaronsteers

Yep, very interesting. I'm not sure I completely agree with the premise that data scientists can get away without knowing anything about underlying compute concepts (at least for now), but I agree wholeheartedly that infrastructure concerns currently take up too much of a data scientists time, and that better tooling is part of the solution. There are recent parallels in adjacent data disciplines - Snowflake makes it possible to use a data warehouse without knowing anything about OLAP architectures (or sorting, vacuuming, distribution strategies etc.) and EMR/Databricks help you to use Spark without needing to be an expert in MPP systems. I believe that packaging Kubernetes in a way that it is convenient for data folks to use without necessarily needing to interact with it directly is a key opportunity for Meltano 🙂