u/alameenswe

Hello so I make that a year working remotely in Nigeria , but I’m split it’s new and I’m in 300l Lagos state university.

Idk what to do , on one hand I’m thinking of going to Canada to further my study or finish here .

Secondly build a life here (houses , cars etc) or move to Canada

Any Canadian in the house to report how the living condition is there? Or someone who’s diaspora

(Pardon any grammatical mistake iPhone’s keyboard is bugging) also to note I work in tech as a swe so yeah .

I woke up this morning to a Cloudflare bill I cannot pay.

$35,000. For a side project with 81 users.

Here's the full story of what happened, how I found it, and what I fixed because I spent 6 hours debugging this and you should never have to.

The setup

I'm building RetainDB a memory layer for AI agents. You send it a conversation, it extracts structured memories, stores them, and lets you search them later. The architecture is Cloudflare Workers + KV + Durable Objects + Queues.

It's been running fine for months. Then last month's bill arrived.

KV Write Operations:      3.13B     $15,635
KV Read Operations:       16.62B    $8,306
DO Storage Rows Written:  4.01B     $3,962
KV List Operations:       574M      $2,870

I have 81 users. That's 350,000 API requests per user per day. I thought I'd been hacked.

I hadn't been hacked.

Bug #1: The infinite queue loop ($15k)

My architecture: user calls /v1/memory → gets queued → ingest worker processes the queue message → ingest worker calls /v1/memory internally to do the actual write.

The ingest worker was passing the original request's write_mode through to the internal call:

js

write_mode: message.write_mode || "direct_write",

When users called the API with write_mode: "async" (the default), the queue message stored "async". The ingest worker then called the API worker with write_mode: "async". The API worker saw async, re-queued it, and returned 202.

The ingest worker marked the job complete.

A new queue message now existed with the same content but a new job ID. The ingest worker processed it. Called the API worker. Got re-queued. Repeat.

Every single async memory write was looping through the queue until the idempotency key eventually deduplicated it — but not before generating 5-10 queue round trips and dozens of KV writes each time.

The fix was one line:

js

write_mode: "sync", // always force sync on internal calls

Bug #2: 4 billion Durable Object writes ($4k)

Every memory write triggered this path through my pending overlay system:

Event	DO storage.put() calls
Enqueue (session scope)	2
Enqueue (user scope, V2 enabled)	2
Ingest: setJobState("processing")	2
Ingest: setJobState("completed")	2
Ingest: ack session scope	2
Ingest: ack user scope	2
Total	12

12 unbatched storage.put() calls per memory write. No batching. No debouncing. At 334 million memory writes per month (driven partly by bug #1), that's 4 billion DO storage writes.

The fix: removed all DO writes from the ingest worker entirely. The pending overlay has a 30-second TTL — it expires on its own. The acks were redundant. The job state DO mirror was redundant (KV already has it). Dropped from 12 to 2 DO writes per memory write.

Bug #3: KV list scan on every request ($2.8k)

API key auth had a 3-step fallback:

1. Hash lookup (1 KV read) ✓ fast
2. Prefix lookup (1 KV read) ✓ fast
3. Full kv.list() scan of all API keys if both miss

Step 3 was running on 95% of requests because the hash/prefix indexes weren't populated for legacy keys. 574 million requests × 1 list scan = 574 million KV list operations at $0.005/1000.

The fix: one flag.

LEGACY_API_KEY_SCAN_ENABLED = "false"

The compounding math

None of these bugs would have been catastrophic alone. Together:

• Bug #1 multiplied every write by 5-10x through queue loops
• Bug #2 multiplied every write by 12x in DO operations
• Bug #3 added a list scan to every single request regardless

81 users → looks like 350k requests/user/day → actually ~30k real requests/user/day amplified 10x.

What I learned

Never pass user-facing write modes through to internal queue workers. The queue consumer IS the async handler. Its internal calls should always be sync.

Durable Object storage.put() is not cheap at scale. Treat it like a database write, not an in-memory assignment. Batch everything. Use TTLs instead of explicit deletes.

Any fallback that touches KV list runs on every request in practice. KV list is $5/million. If your auth fallback does a list scan, it will do it on every cold request.

Set up Cloudflare spending alerts before you need them. There's no hard spending cap on Workers. I found out about this from the bill, not an alert.

The fixes are deployed. The bill is sent to Cloudflare support with a full explanation. The product still has 81 users and is still running.

If you're building on Cloudflare Workers and Durable Objects audit your DO write patterns before you ship. Especially if you have any queue consumer that calls back into your own API.

Happy to answer questions. Yes I'm not okay. No, I don't know if Cloudflare will credit it.