r/allenai

Researchers at UT Austin, Northeastern, and MD Anderson used our fully open Olmo 3 to investigate whether LLMs actually know specific drugs—or infer from patterns in their names.

For 51–59% of tested drugs, Olmo 3 showed little evidence of drug-specific knowledge. Another 12–18% appeared driven by affixes like “-pril” or “-olol,” which can reveal a drug’s class.

Because we release model weights, training data, documentation, and intermediate checkpoints, the researchers could trace the behavior further. Using our infini-gram engine for searching massive text corpora, they found that drugs appearing less often in training were more likely to trigger these naming shortcuts.

It’s a useful example of what fully open models enable: not just spotting a model behavior, but investigating where it comes from.

Read more: https://allenai.org/blog/olmo-drug-morphology

Today we're introducing a preview of TutorMoments, a framework that measures whether AI tutors can make one of the hardest calls in teaching: when to step in and help a student, & when to hold back and let them do the heavy thinking. 👇

Language models are trained to be helpful, and a helpful assistant tends to do the hard part of learning for you: explains the concept, lays out the steps, & guides you to the answer. That can cut short the productive struggle that leads to stronger understanding.

TutorMoments is built on transcripts of real one-on-one math tutoring. We had experienced teachers read them & flag key moments—decision points where the tutor had to choose between making a problem easier & pushing the student to do more of the reasoning.

TutorMoments pauses a transcript at these key moments & lets an LLM take over as the tutor; another model stands in for the student. Each replay is scored: did the tutor support the student when needed, push for harder thinking when they were ready, & avoid over-helping?

In our replays, models told only to "tutor well" tend to over-help, providing lots of support but rarely pushing toward deeper thinking. That suggests a model's default helpful-assistant behavior isn't enough on its own to tutor well.

Spelling out the trade-off in the prompt helps—every model we tested scores higher once told when to help vs. when to hold back. But it only goes so far. Models still differ widely in how reliably they make that call, & even the best scorers have plenty of room to improve.

To gather feedback, we're releasing de-identified annotated tutoring transcripts plus code & scored replays.

💻 Code: https://github.com/allenai/tutormoments

🤗 Data: https://huggingface.co/datasets/allenai/tutormoments-preview
🌐 Learn more: https://allenai.org/blog/tutormoments