Wanderer's Tale - The Missing Quote Mark That Broke 186 Items
Two nights in a row, my game server crashed. Then the admin page went down too. I finally traced it to one missing punctuation mark — copied 186 times across the database.
Here's what happened, for anyone who likes a good bug story.
Somewhere in the tooling I use to add items to the game, stat bonuses were getting saved in the wrong format. Instead of proper JSON like "strength": 2 with quotes around the key, they were stored the way I'd write it in my code — strength: 2, no quotes.
Small difference, big consequence. The code that reads that data can't parse the unquoted version, so every time the game loaded one of those items, it quietly gave up and acted like the item had no bonuses at all. People were running around in gear that did absolutely nothing, and neither they nor I had any idea.
I fixed it in two passes.
First, I stopped the crashing. I wrapped every database read so a single bad row can't take the whole server down anymore. That ended the 2am wakeups.
Then I went after the actual data, because those broken values were still sitting there, silently zeroing out everyone's gear. I wrote a repair script to find every value that failed to parse and rewrite it properly.
Here's the part that got me. I expected to fix one item. I found six. And then, because I decided to scan the rest of the database instead of calling it a day, I found 180 more broken rows with a completely different formatting problem I didn't even know I had. If I'd only patched the thing that was crashing, those would still be quietly broken right now.
The lesson I keep relearning working on this alone: when you find one instance of a data bug, assume there are more and go count. The bugs that scare me aren't the loud ones that crash the server. They're the quiet ones that fail politely and hand back a number that looks completely reasonable.
All 186 rows are fixed and verified. Gear does things again.
I've learned the hard way that a fix isn't really done until the problem can't crawl back...I hope.
The Wanderer - Wanderer's Tale