u/wildwarrior007

Hey everyone, looking for some architectural advice on handling Redis failovers gracefully under high traffic.

Our Setup:

Node.js backend using ioredis

Redis Sentinel (Bitnami Helm Chart) running on AWS EKS (Karpenter for node provisioning)

1 Master, 2 Replicas

What we've done so far: We found that the default Bitnami preStop hook uses CLIENT PAUSE during pod termination, which freezes our app for ~20s and causes massive TimeoutErrors.

We overwrote the preStop script to remove CLIENT PAUSE and instead trigger a SENTINEL FAILOVER immediately, followed by cleanly severing the TCP connections. On the Node.js side, we use ioredis with maxRetriesPerRequest: null and enableOfflineQueue: true.

The Result: When a node is drained, ioredis catches the dropped connection, buffers all incoming commands in memory, asks Sentinel for the new master, and flushes the queue once connected. The failover usually takes about 2 to 5 seconds. To the end user, this just looks like a slightly slower API request. No 500 errors.

My Questions for the community: While this works perfectly in testing, I know we can't guarantee a strict 2-second failover in production.

Under heavy traffic and large datasets, Sentinel elections and DNS propagation could easily push this delay to 5-10 or 15 seconds or more.

If the delay extends to 10 seconds under massive traffic, our Node.js ioredis in-memory buffer will explode in size, potentially causing OOM crashes on the application side, or massive latency spikes when it finally flushes thousands of queued commands to the new master at once.

How do you handle this at scale?

Do you just accept the 5-10 second latency spike during a failover?

Is migrating to a managed service like AWS ElastiCache the only way to avoid this completely?

Would love to hear how folks are handling Redis HA edge cases at scale!

Hey everyone, looking for some architectural advice on handling Redis failovers gracefully under high traffic.

Our Setup:

Node.js backend using ioredis

Redis Sentinel (Bitnami Helm Chart) running on AWS EKS (Karpenter for node provisioning)

1 Master, 2 Replicas

What we've done so far: We found that the default Bitnami preStop hook uses CLIENT PAUSE during pod termination, which freezes our app for ~20s and causes massive TimeoutErrors.

We overwrote the preStop script to remove CLIENT PAUSE and instead trigger a SENTINEL FAILOVER immediately, followed by cleanly severing the TCP connections. On the Node.js side, we use ioredis with maxRetriesPerRequest: null and enableOfflineQueue: true.

The Result: When a node is drained, ioredis catches the dropped connection, buffers all incoming commands in memory, asks Sentinel for the new master, and flushes the queue once connected. The failover usually takes about 2 to 5 seconds. To the end user, this just looks like a slightly slower API request. No 500 errors.

My Questions for the community: While this works perfectly in testing, I know we can't guarantee a strict 2-second failover in production.

Under heavy traffic and large datasets, Sentinel elections and DNS propagation could easily push this delay to 5-10 or 15 seconds or more.

If the delay extends to 10 seconds under massive traffic, our Node.js ioredis in-memory buffer will explode in size, potentially causing OOM crashes on the application side, or massive latency spikes when it finally flushes thousands of queued commands to the new master at once.

How do you handle this at scale?

Do you just accept the 5-10 second latency spike during a failover?

Is migrating to a managed service like AWS ElastiCache the only way to avoid this completely?

Would love to hear how folks are handling Redis HA edge cases at scale!

Hey everyone, looking for some architectural advice on handling Redis failovers gracefully under high traffic.

Our Setup:

• Node.js backend using ioredis
• Redis Sentinel (Bitnami Helm Chart) running on AWS EKS (Karpenter for node provisioning)
• 1 Master, 2 Replicas

What we've done so far: We found that the default Bitnami preStop hook uses CLIENT PAUSE during pod termination, which freezes our app for ~20s and causes massive TimeoutErrors.

We overwrote the preStop script to remove CLIENT PAUSE and instead trigger a SENTINEL FAILOVER immediately, followed by cleanly severing the TCP connections. On the Node.js side, we use ioredis with maxRetriesPerRequest: null and enableOfflineQueue: true.

The Result: When a node is drained, ioredis catches the dropped connection, buffers all incoming commands in memory, asks Sentinel for the new master, and flushes the queue once connected. The failover usually takes about 2 to 5 seconds. To the end user, this just looks like a slightly slower API request. No 500 errors.

My Questions for the community: While this works perfectly in testing, I know we can't guarantee a strict 2-second failover in production.

1. Under heavy traffic and large datasets, Sentinel elections and DNS propagation could easily push this delay to 5-10 or 15 seconds or more.
2. If the delay extends to 10 seconds under massive traffic, our Node.js ioredis in-memory buffer will explode in size, potentially causing OOM crashes on the application side, or massive latency spikes when it finally flushes thousands of queued commands to the new master at once.

How do you handle this at scale?

• Do you just accept the 5-10 second latency spike during a failover?
• Is migrating to a managed service like AWS ElastiCache the only way to avoid this completely?

Would love to hear how folks are handling Redis HA edge cases at scale!

We're running a processing service on EKS with KEDA and are trying to find the best way to handle CPU-intensive workloads without wasting compute.

Current setup:

• KEDA minReplicaCount = 3
• requests.cpu = 1
• limits.cpu = 6
• Service is mostly idle but during video processing it can use close to 6 CPUs
• Nodes in the cluster include some 4 vCPU instances

The problem is that Kubernetes schedules based on requests, so pods can land on a 4 vCPU node because they only request 1 CPU. When a video processing workflow starts, the pod tries to consume up to 6 CPUs and we start seeing CPU saturation/throttling alerts.

If I increase requests.cpu to 5 or 6, scheduling becomes more accurate, but then I'm permanently reserving that capacity for pods that spend most of their life idle, which feels wasteful.

Questions:

1. How are others handling bursty CPU-heavy workloads like video processing?
2. Do you set requests=limits for these worker pods?
3. Do you separate API pods from processing workers?
4. Are you using Karpenter or Cluster Autoscaler to provision larger nodes on demand?
5. Is there a common pattern for EKS + KEDA + video processing workloads that avoids both CPU starvation and idle compute costs?

Would appreciate hearing how people are solving this in production.