Five Clusters. Five Lessons. One Production System.
I've been running self-hosted Kubernetes in production for five years. Not managed EKS or GKE, actual bare metal nodes I provisioned myself. Over that time I rebuilt the cluster five times. Each time because the previous version couldn't solve a specific problem.
Here's what actually drove each decision:
Stage 1 - needed HA. Followed Techno Tim's k3s guide. Used three DigitalOcean VPS running nginx as a makeshift load balancer with a cloud LB in front. It worked but the most expensive components were doing the least important work.
Stage 2 - RAM was sitting idle on Hetzner nodes. Moved to Contabo. Contabo had no private network at the time so I built a WireGuard mesh with Netclient. Removed the nginx VPS and cloud LB entirely. Klipper replaced them.
Stage 3 - Oracle Cloud ARM nodes are free. Extended the WireGuard mesh to include them as workers. Used GoReleaser to build multi-arch images via GitHub Actions. Master nodes stayed on amd64, workers on ARM64.
Stage 4 - didn't want port 80 and 443 open on every node. Tried Calico BGP with MetalLB to announce a private LB IP. Architecturally correct. Ran it for a month. HTTP latency was noticeably high. Reverted. Kept the internode mesh, went back to Keepalived for the floating IP.
Stage 5 - I had never successfully run a firewall alongside Kubernetes without it with CNI. Saw Cilium at KubeCon. Cilium's host firewall runs at the eBPF layer below where CNI conflicts happen. Moved to rke2 + Cilium on OVH bare metal. Every node is now egress only on the public interface. Cloudflare-only ingress with mTLS. Hard tenant isolation between namespaces by default.
The MongoDB crashloop ghost that followed me through three different providers and two CNIs also mysteriously stopped at Stage 4. Never diagnosed. Just gone.
I wrote a full write up with architecture diagrams at each stage
Curious if anyone else has hit unexpected latency when routing traffic through an additional network layer on a self-hosted setup.