u/Ok-Accountant-6783

I made an anonymous comeback in my department

I joined as a mid level engineer and during my first architecture review, I noticed the caching layer was writing user profile updates to a local cache and syncing to the database asynchronously. If two requests hit different app servers within the sync window, the second write would overwrite the first. 

I raised it in the review and the staff engineer who designed the system said "the sync window is 50 milliseconds, the probability of two profile updates from the same user hitting different servers in 50ms is negligible." I said "it's not negligible if you have a form that auto saves on every field change” and our profile page had 11 fields. A user tabbing through the form and editing 3 fields would fire 3 PUT requests within 200m and with round robin load balancing, they'd likely hit different servers.

He said I didn't understand how the load balancer worked and I seriously dropped it. I was new but he had 14 years of experience and I assumed I was missing something.

Six months later (last week) support got a cluster of tickets. Users said their profile changes were "reverting." A user would update their phone number and address, come back the next day and the phone number was correct but the address was old or the other way around sometimes. The staff engineer debugged it for 2 weeks and the root cause was exactly what I'd described haha. 

He fixed it with a database level optimistic lock but he never mentioned my original comment and I didn't bring it up either.

reddit.com
u/Ok-Accountant-6783 — 13 days ago