Best k8s for on premise production
Hi everyone,
I’ve been assigned a task to architect a highly available and fault-tolerant on-premises Kubernetes cluster for GPU-based workloads, with a target of around 1,000 users (50 users /day) with time slicing enabled.
I’d like to get your recommendations on which Kubernetes distribution would be the best fit for a production-grade environment with minimal day-to-day operational overhead.
Currently, we have a MicroK8s-based cluster with Longhorn, running Kubeflow through Charmed Kubeflow. However, we are facing recurring operational issues, such as Calico failures, pods unexpectedly becoming pending, and other cluster-level instability.
Given these requirements, I’d like to understand:
- Which Kubernetes distribution would be the best fit for this environment?
- What would be the recommended HA and fault-tolerant architecture for GPU workloads?
- Which solution would provide the lowest operational overhead while maintaining production-grade reliability?
- Are there any recommended alternatives to MicroK8s + Longhorn + Charmed Kubeflow for this scale?
Any recommendations or real-world experience would be greatly appreciated.