Am I wrong that a 429 shouldn't count against a job's retry budget?
Been going back and forth on this one and I want to know if I'm alone in it.
Most retry implementations I've seen (BullMQ, hand-rolled wrappers, most of the managed stuff) treat every failed attempt identically. Job comes back non-2xx, attempt counter increments, backoff applies, after N attempts it's in the DLQ.
But a 429 isn't a failure, nothing broke. The downstream is telling you exactly when to come back and usually handing you a Retry-After header to do it with. If you burn an attempt on it, sustained rate limiting at a provider will walk perfectly good jobs into the DLQ while your actual error budget (the one meant for 500s, timeouts, connection resets) never gets spent on what it's for.
So I've been treating 429/503/529 as a defer rather than a failure: honor Retry-After, requeue, don't decrement. Works, but it opens two things I don't have clean answers to.
First, you need a ceiling or the queue never drains. A provider that 429s indefinitely will requeue that job forever. I've landed on two different ceilings: a wall-clock deadline (dead 24h after it's due, regardless of how it got there) and a separate max defer count. Blowing the defer ceiling dead-letters the job under its own reason rather than folding it into "out of retries" which matters because those are different failures. One says the downstream is broken, the other says it's been unusable long enough that it may as well be. At a certain point temporarily unusable === broken.
Second, deferred jobs are invisible. They aren't failing, so they don't trip anything you're monitoring, and you can sit on a queue that isn't draining and looks completely healthy. Feels like deferred jobs need their own state and their own alerts rather than being folded into "pending" or "processing".
Anyone handling this differently? Specifically curious whether people distinguish 503 from 429, I lump them together, but 503 is ambiguous in a way 429 isn't.