10 learning modes, students used them all, then 80% quit 5-6 of them by week 3
We shipped 10 learning modes. Students used them all in the first week. Then most of them stopped using most of them, and we genuinely don't know why, and our metrics don't actually tell us anything useful. That's where I'm sitting right now.
The usage pattern is pretty consistent: a new student activates somewhere between 7 and 8 modes in their first week, which felt like a win when we first saw it. Exploration, curiosity, exactly what you'd want. But by week 3, the same students are down to 2-3 modes, sometimes just 1. And the ones that survive aren't the ones we designed to be the pedagogical heavy-lifters. Flashcards stuck around way more than we expected. Socratic-style doubt-solving, which we thought was the most sophisticated thing we'd built, gets dropped almost as often as the novelty modes we honestly weren't that confident about. I've been staring at these logs trying to figure out if the survivors stuck because they're actually producing learning or because they're just frictionless and habit-forming in a Duolingo-streak kind of way. Those are very different things.
I don't know if there's a clean way to separate pedagogical utility from engagement theater without running something close to actual controlled experiments, which is hard when your users are K-12 students and the outcome you care about is internalization, not session time. The proxy metrics we have, completion rate, return rate, time-on-mode, all reward the wrong thing. A student who opens flashcards for 4 minutes every day looks identical in our logs to a student who's genuinely drilling retrieval and self-testing. A student who spends 20 minutes on a mock test might be learning or might just be more anxious. We built this into an AI tutoring platform ([padhaao.in](https://padhaao.in/), I'm behind it) and I'm still not confident we got it right, or that we even asked the right questions before shipping.
The question I keep coming back to is: what would actually distinguish a feature that changes how a student internalizes knowledge from one that just feels comprehensive to a parent looking at the feature list? Because some of these modes might genuinely exist for the sales page and not for the student, and I'm not sure we were honest enough with ourselves about which was which when we scoped them. One heuristic I'm starting to think has some teeth: does the mode require the student to generate something (an answer, a connection, a prediction) or just receive something? Modes that are purely receptive might be stickier in the short run but probably aren't doing as much. Could be wrong about this, and I'm aware that's an oversimplification of a lot of cognitive load research. But it's the frame that's making the most sense to me when I look at which modes are actually surviving past week 3.
What I haven't figured out is how to instrument this properly without basically running a longitudinal study on students who are actively trying to pass board exams. The stakes are real for them and I'm not going to mess with their study routines to satisfy my curiosity about feature retention. So I'm asking here because I don't have a good answer and I'd rather hear from people who've thought about this more carefully than I have: how do you measure whether a pedagogical feature is doing real work vs. just existing?