u/LearnWorlds_

We all know the drill, AI can now generate a full training course in a few minutes. In a few years, it will probably even manage complex branching scenarios well.

But it begs the question, are we're tracking the wrong thing? Yes, a course exists now, or a quiz got completed. That feels like progress, and it's structured and efficient enough that it's easy to mistake for real progress. Except watching an AI-generated module, or passing its auto-graded quiz, doesn't actually prove anyone can do the job better under pressure. All we really did was swap a content metric for a capability metric and called it done.

The worrying part is how this compounds. If a company leans on AI to generate junior-level work and also leans on AI to generate the training that used to build junior people into senior ones, you can lose the pipeline without anyone deciding to. Three years later you're short on the people who were supposed to grow into the harder problems.

This is a conversation we've been having internally. Has anyone actually measured this at their org? Not "did people complete the AI-generated course" but whether people who went through AI-generated training perform differently on the job than people who didn't?

A founder spoke about this at our recent AI summit, and wanted to share here to get other industry folks' thoughts.

He built an AI chatbot for frontline manager training which was meant to help new managers practice the hard stuff like budget conversations, hiring decisions, difficult 1:1s so they could better manager their teams.

But when he read through the anonymized transcripts of what people were asking it, it turned out most of them weren't rehearsing conversations with their teams. They were using it to manage up instead of to manage their teams. The log showed people were actually rehearsing conversations with their bosses, floating questions like how to ask for more headcount, how to pitch a budget so it doesn't get cut, etc.

Basically they were using the tool in a completely different way than it was intended.

His point was that no survey or LMS completion report would ever have caught that. You can track whether someone finished a module, but you can't track what they were actually anxious about the night before a budget meeting. The chat log did, because people say things to an AI they'd never put in a feedback form.

It pretty much sums up a common issue in instructional design. There seems to be a big gap in what IDs design training for versus what people actually need it for.