I watched 50 session recordings of new users. 19 of them were using a product I'd never seen.
I installed Hotjar because I wanted to see where new users got stuck in onboarding.
What I expected: confusion about features and unclear copy. What I actually found: 19 out of 50 users were experiencing a product that looked nothing like mine.
Buttons overlapping on smaller screens. A dropdown menu that wouldn't open on Safari. I'd been building and testing on a 16-inch MacBook Pro with fast internet. These users were on Chromebooks and old iPads with spotty connections. The product behaved differently for them in ways I couldn't have predicted.
The worst part: I'd been reading my analytics thinking "onboarding completion at 62%, not bad." But that 62% was the surviving users. The other 38% were hitting bugs I couldn't see from my own setup.
I spent a week testing on cheaper hardware and slower connections. Fixed 11 issues. Onboarding completion went from 62% to 79%.
You can't improve an experience you haven't actually seen. And unless you're testing on the devices your users actually own, you probably haven't seen what most of them see.