Docker Hub rate limit reached during K8S upgrade, best practices?
We're running into Docker Hub rate limiting during Kubernetes upgrades and I'm curious how others solve this at scale.
Let's say you have 100+ containers coming from external registries (mostly Docker Hub images like busybox, alpine, utility sidecars, etc.).
During a Kubernetes upgrade or large node rotation, eventually new pods start failing with errors like:
Init:failed to pull and unpack image "docker.io/library/busybox:1.37.0": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/busybox/manifests/sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e: 429 Too Many Requests - Server message: toomanyrequests: You have reached your unauthenticated pull rate limit.
The 101st image pull basically kills the rollout.
I'm interested in how people operating larger clusters handle this in practice.Some options I can think of:
- configuring imagePullSecrets everywhere
- using dedicated ServiceAccounts with registry credentials
- mirroring all external images into an internal/private registry
- registry pull-through cache (Harbor, Artifactory, Nexus, etc.)
- pre-pulling images onto nodes
- completely avoiding Docker Hub in production
What has worked best for you operationally?
—-
EDIT: The K8S is an AKS