https://thundergolfer.com/kubernetes/infrastructure/data-engineering/2021/11/04/from-data-eng-to-sys-admin-put-down-k8s/ The last part is great:

I think the key features of this better future would be:

• A K.I.S.S cron scheduler service. It baffles me that Google Cloud and AWS make cron scheduling a container so annoying. Read ‘Creating and configuring cron jobs’ and it seems sensible until you get to “job’s target” and it doesn’t list a fucking container as an option_._ • Resources, not VMs. Don’t make users pick instance types. The default K8s

request

and

limit

user interface is fine, actually. A company’s data engineers just shouldn’t have to wrangle the pile of VMs facilitating that interface. • Data architecture best-practices made obvious. K8s made it hard to build non-stateless applications, because it came in with the wisdom that stateless is the way. For data, we have similar wisdom with which to imbue our platform systems. ◦ Data batch jobs can be mostly modelled as a transform function over some structured input data, so the batch job platform should encourage that kind of problem decomposition. ◦ Idempotency should be a first-class concept, as well as dataflow. ◦ Be opionated about best-practices like dry-run testing, staging areas. • Don’t wrap the cloud system in so much YAML and cloud services minutae that a local-prod impedence mismatch destroys our tight dev loops. I should not need to run a mini-container orchestrator on my laptop to test my pipeline changes.

Interesting read. My first impression is that this is a tooling maturity problem (which is why we exist 😅). I think the real wisdom here is:

without concerted effort on the part of an infrastructure engineer to hide K8s from the product engineer, that person finds themselves floundering without K8s knowledge

Applying that "concerted effort" to insulting data practitioners from the complexities of k8s rather than shunning the technology altogether. I see parallels with databases pre-RDS. Every org needed a DBA, and dev's complained about complexity spill-over. Now companies rarely need DBA expertise; AWS handles the operational complexity, and ORMs in all the common frameworks handle routine administration (creating tables etc) in a developer-friendly way. RDS was released in 2009, k8s in 2014. It still has some maturing to do 🙂