u/samarpitvatry

We shipped 10 learning modes. Students used them all in the first week. Then most of them stopped using most of them, and we genuinely don't know why, and our metrics don't actually tell us anything useful. That's where I'm sitting right now.

The usage pattern is pretty consistent: a new student activates somewhere between 7 and 8 modes in their first week, which felt like a win when we first saw it. Exploration, curiosity, exactly what you'd want. But by week 3, the same students are down to 2-3 modes, sometimes just 1. And the ones that survive aren't the ones we designed to be the pedagogical heavy-lifters. Flashcards stuck around way more than we expected. Socratic-style doubt-solving, which we thought was the most sophisticated thing we'd built, gets dropped almost as often as the novelty modes we honestly weren't that confident about. I've been staring at these logs trying to figure out if the survivors stuck because they're actually producing learning or because they're just frictionless and habit-forming in a Duolingo-streak kind of way. Those are very different things.

I don't know if there's a clean way to separate pedagogical utility from engagement theater without running something close to actual controlled experiments, which is hard when your users are K-12 students and the outcome you care about is internalization, not session time. The proxy metrics we have, completion rate, return rate, time-on-mode, all reward the wrong thing. A student who opens flashcards for 4 minutes every day looks identical in our logs to a student who's genuinely drilling retrieval and self-testing. A student who spends 20 minutes on a mock test might be learning or might just be more anxious. We built this into an AI tutoring platform ([padhaao.in](https://padhaao.in/), I'm behind it) and I'm still not confident we got it right, or that we even asked the right questions before shipping.

The question I keep coming back to is: what would actually distinguish a feature that changes how a student internalizes knowledge from one that just feels comprehensive to a parent looking at the feature list? Because some of these modes might genuinely exist for the sales page and not for the student, and I'm not sure we were honest enough with ourselves about which was which when we scoped them. One heuristic I'm starting to think has some teeth: does the mode require the student to generate something (an answer, a connection, a prediction) or just receive something? Modes that are purely receptive might be stickier in the short run but probably aren't doing as much. Could be wrong about this, and I'm aware that's an oversimplification of a lot of cognitive load research. But it's the frame that's making the most sense to me when I look at which modes are actually surviving past week 3.

What I haven't figured out is how to instrument this properly without basically running a longitudinal study on students who are actively trying to pass board exams. The stakes are real for them and I'm not going to mess with their study routines to satisfy my curiosity about feature retention. So I'm asking here because I don't have a good answer and I'd rather hear from people who've thought about this more carefully than I have: how do you measure whether a pedagogical feature is doing real work vs. just existing?

Day 1 a student asks why entropy increases in a free expansion. I explain it. Day 3 the same student asks why the universe tends toward disorder. Day 8 they ask why you can't just reverse a gas expanding into a vacuum. Same concept, three different entry points, and honestly I didn't connect them until I looked back at the chat history.

What I still can't figure out is whether that's a tutoring failure, a platform problem, or just how that particular brain needed to approach the idea, circling it from different angles until something clicked. Because the outcome was fine. The student got it eventually. But I have no idea what actually helped.

I've been thinking about this a lot because the students I support come through platforms that offer both AI doubt-solving and live explanations. [padhaao.in](https://padhaao.in/) is one I work with a lot, and the triage question comes up constantly: send this to the AI or escalate it. What I've landed on, tentatively, is three rough categories. Knowledge gap: the student doesn't have the prerequisite, they need a better explanation or a different one. That's usually AI-fine if the explanation is good. Lookup problem: they just need the formula or definition and they know what to do with it. Obviously AI. And then there's a third thing that I don't have a clean name for, the student who understands every individual piece but can't hold the whole thing together, or the one who needs someone to tell them it's okay to not get it yet before they can actually try. That third category is where human tutors earn their existence and where I see AI fall flat not because the explanation is wrong but because the student wasn't in a state to receive an explanation at all.

The counterintuitive part is that the third category doesn't announce itself. It often looks like a knowledge gap. The student asks a content question. You answer it. They ask again, differently. You answer again. Nothing lands. And the thing that finally helps isn't a better explanation, it's asking them what they think the answer is first, or asking where exactly it stops making sense to them, or sometimes just sitting in the confusion with them for a minute.

So the diagnostic questions I've started using before I decide what to do: can the student tell me what they already tried? Can they point to the specific moment the logic breaks down for them, not just say "I don't get it"? If they can do both, it's probably a knowledge or lookup problem and an AI explanation will work. If they can't, if the confusion is undifferentiated, that's usually the metacognitive thing and they need a person.

I don't know how to get students to tell me this without asking directly, which is its own problem. And I genuinely don't know how to predict it before they ask anything at all.