u/herchila6

This one bit a service of mine and it's worth a PSA, because every signal you have lies to you while it's happening.

The scenario: a Celery worker loses its Redis connection for a moment (managed Redis idle timeout, a network blip, a reset). Celery logs it, once:

consumer: Connection to broker lost. Trying to re-establish the connection...

And then it just... doesn't. The process stays alive, your orchestrator shows it green, redis-cli ping works fine, and no exception ever reaches your app. But the worker never consumes another task. The queue grows in silence until somebody restarts the container.

If you think this is ancient history, two reports from May 2026 that are still open:

• celery/celery#10303 — worker on Azure Redis takes a Connection reset by peer, logs the reconnect attempt, never recovers. Reporter says it was "fixed" in 5.6.3, upgraded, still hits it.
• celery/celery#10325 — DigitalOcean Managed Redis, stalls for exactly the idle-timeout interval, reproduced on 5.4.0 / 5.5.3 / 5.6.3 / main. The part that got me: they already had health_check_interval, socket_keepalive and retry_on_timeout set, and it stalled anyway. Why nothing catches it: no exception, so error-rate alerts stay quiet. Process alive, so the liveness probe passes. Redis healthy, so a broker check passes. The only thing that moves is queue depth creeping up, which is the one noisy signal most teams never alert on.

Settings that help (set them, but they are not a guarantee, see #10325):

• broker_transport_options with socket_keepalive + health_check_interval ~25
• broker_connection_retry_on_startup=True, broker_connection_max_retries=None
• worker_cancel_long_running_tasks_on_connection_loss=True (becomes the default in 6.0) The thing that actually catches it: stop monitoring "is the worker up" and start monitoring "did a task complete recently." Poll inspect active / inspect stats, track the last-completed timestamp per worker, and alert when a worker goes quiet while its queue is non-empty. Active empty + queue not empty = ghost.

Anyone else run into this? Curious how people detect it beyond a queue-depth alarm, especially on managed Redis.