That's a wrap on #COLM2026! Thank you to everyone who stopped by our booth, joined our evening gathering, & talked open research with us.
Catch up on the papers, interviews, & highlights from the week. 👇
We’re sharing the engineering work behind Ai2’s GPU scheduler—our new approach to allocating compute across research teams. 👇
On our largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds.
Under our old scheduler, every scheduled workload eventually used HIGH priority. Some researchers kept idle jobs running to reserve GPUs for experiments. The incentives rewarded holding on to capacity even when there were no active jobs.
Managers now assign GPU-time budgets to research programs & projects. Our fair-share scheduler compares recent use with those budgets, moving work from underused allocations ahead in-queue.
Jobs declare the minimum runtime needed to make progress. Budgeted work is protected until then. Afterward, it can keep running or be interrupted; resumable jobs return to the queue.
Over a 30-day test with our new scheduler, teams received 98% of the GPU hours they were owed, accounting for actual demand. Cluster occupancy stayed at 98%, with spare capacity available to interruptible work that didn’t draw down a team’s budget.
📝 Read more about the design decisions behind our scheduler, rollout lessons, & what we’re tackling next: https://allenai.org/blog/impactful-scheduling
We're co-hosting Open Together with Hugging Face!
Join the open AI community at The Midway in San Francisco on Oct 16 to kick off Open Source AI Week. Help us build a future where AI is open, secure, and fully transparent.
The event is free. RSVP: https://luma.com/OpenTogether
Introducing AstaBrief 8B, an open model that turns complex research questions and literature excerpts into cited reports.
Run it locally on your own hardware, with open weights and training data you can inspect & build on. 👇
AstaBrief powers Fast mode in Asta’s Generate a report feature. It writes reports in one pass, skipping intermediate steps that summarize & group the retrieved excerpts so researchers can get a preliminary report sooner.
We built AstaBrief on Qwen3-8B using queries from ScholarQA & our earlier literature-synthesis systems. We removed bot/test traffic & very short prompts and used an LLM to filter non-English or non-scientific requests & prompts with personal info, leaving 90K queries.
From selected queries in the 90K pool, we generated cited reports. Quality filtering left 39.5K examples for supervised fine-tuning (SFT). We used separate queries to build ~6K report pairs where two judge models agreed, then trained with direct preference optimization (DPO).
AstaBrief is built for speed & quality. Across Asta’s full pipeline, Fast mode averages 51.1s per report vs. Claude-powered Thinking mode’s 178.5s—about 3.5× faster. And in our evaluations, AstaBrief is competitive with Thinking mode & DR Tulu on answer + citation measures.
Our example repo provides a starting point for generating reports from your own PDFs locally. For more info, read our blog.
🤗 Download AstaBrief 8B: https://huggingface.co/allenai/AstaBrief_8B
🌐 Try Fast mode: https://asta.allen.ai/
💻 Example repo: https://github.com/allenai/ai2-scholarqa-lib/tree/main/api/scholarqa/lite
📊 Training data: https://huggingface.co/collections/allenai/astabrief
📝 Blog: https://allenai.org/blog/astabrief
We’re releasing Olmo-core 3—open training infrastructure for large mixture-of-experts (MoE) models. 👇
It’s a core system behind the next generation of Olmo, designed to scale into the trillion-parameter range.
MoEs use only some of their specialized components – or experts – for each token. That saves computation, but storing the full model & routing data across GPUs can erode those savings as models grow.
Olmo-core 2 repeatedly gathered model weights for each small batch of training data. Olmo-core 3 keeps experts resident on GPUs & sends data to them, avoiding that repeated weight gathering.
Olmo-core 3 maintains nearly the same training speed across MoEs of increasing total size, w/ roughly the same number of parameters active per token. It delivers about 2.7× the throughput of Olmo-core 2 for the same model on the same number of GPUs.
Explore the concepts behind Olmo-core 3: https://narrative.allen.ai/scaling-up-training
The tech report covers experiments, findings, & failures, including token gerrymandering, where a load-balancing score could improve as workloads became less balanced: https://allenai.org/papers/olmocore3
Train your own MoEs with Olmo-core 3—adapt the framework to your hardware & experiment with distributed training.
💻 GitHub repo: https://github.com/allenai/olmo-core
📝 Learn more in our blog: https://allenai.org/blog/olmocore3
Google Cloud put Olmo 3’s reproducibility to the test, rerunning our 7B pretraining & mid-training on Cloud TPUs and matching our original run on held-out evals.
Reproducibility matters for science and trustworthy AI. That’s what fully open makes possible. 🤝
Can a fully open model help make AI more “prosocial” through a game? Soham Padia built Steering Arena, where players try to elicit kind & respectful responses from Olmo 3. Surprisingly, strings like “Undert! AH :-) Rog Appl)” were highly effective.
Padia wanted to know what researchers might miss in evals of prosocial AI behavior—whether models respond in helpful, fair, safe, & considerate ways. So he created Steering Arena, where players submit short prompts & compete to steer Olmo 3 toward responses scored as prosocial.
Padia used the National Deep Inference Fabric, an NSF-supported platform for studying large open models remotely, to work with Olmo 3-32B without having to host it himself. Broadening access to advanced fully open AI is also central to our NSF OMAI work: https://allenai.org/blog/omai-compute-now-live
To build Steering Arena, Padia showed Olmo pairs of answers to the same prompts – one more prosocial, one less – and looked for the internal pattern that distinguished them. This gave Steering Arena a way to score how strongly new prompts pushed Olmo toward prosocial responses.
Surprisingly, the best-scoring Steering Arena entries were basically gibberish. After ~600 submissions, all top 36 were strings no person would normally write, like “Undert! AH :-) Rog Appl).” Players could score well without writing text that made sense to a human.
Because we release more than model weights with Olmo, Padia could see what Steering Arena rewarded, investigate why nonsense strings scored so well, & share the measurements behind the scores—so others can improve how the field evaluates prosocial AI behavior.
👉 Learn more: https://allenai.org/blog/olmo-arena
How should future scientists learn to interrogate AI tools for discovery? Earlier this year, University of Washington students put AutoDiscovery, our tool for generating and testing promising scientific leads, to the test in a classroom challenge designed around that question.
The goal wasn’t simply to generate better hypotheses—students had to choose a research question and dataset and scrutinize AutoDiscovery’s outputs. Tasked with investigating where the tool's reasoning broke down, the students had to decide whether unexpected results reflected a strong scientific theory, a problem with the data, or a limitation of the tool itself.
In our new article, we look at what the challenge revealed about teaching scientists to work critically with AI—and why scientific judgment remains central even as these tools become more capable.
Read the full story below, and try AutoDiscovery for yourself. Credits are now extended through Dec. 31. 👇
Training an LLM by showing it answers people prefer – and ones they don’t – can improve it while quietly worsening some behaviors.
Goodfire AI used our open post-training stack to predict how a full training run would change responses to different prompts. 👇
Teams often discover unwanted changes in model behavior only after training is finished. Then they have to work backward from evaluation scores to guess which of hundreds of thousands of training examples caused them.
Goodfire wanted to move that debugging earlier—and Ai2’s open stack made it possible:
• Our Dolci dataset provides Olmo 3’s preference data
• Olmo includes intermediate checkpoints + reproducible recipes
• OLMES measures changes in model capabilities
Using that stack, Goodfire developed predictive data debugging: a way to estimate which behaviors preference training will strengthen or suppress before committing compute to a full run.
In one experiment, preference training improved Olmo’s general capabilities but also made it more likely to answer some harmful requests framed as fiction or hypotheticals.
Goodfire traced part of the regression to specific Dolci preference pairs.
This is why publishing more than model weights matters: researchers can trace unexpected behavior back to the data behind it, test targeted changes to training, and measure whether those fixes work without weakening the model elsewhere.
We're at ECCV 2026 with papers & talks across the conference. Come say hello and learn about our latest research!
ML emulators mimic climate processes faster than traditional models. Next is coupling atmosphere & ocean emulators so their predictions feed into each other as the simulation runs.
With the E3SM team, we built a system that does that: SamudrACE-E3SMv3.
SamudrACE-E3SMv3 is a fully coupled atmosphere-ocean emulator that reproduces the pre-industrial control run of E3SMv3, the Earth system model it learns from. E3SM simulates ~28 years of climate per day on 105 compute nodes versus the emulator’s ~1,100 years/day on one H100 GPU.
Training is staged. The atmosphere (ACE) & ocean (Samudra) emulators pretrain separately on "perfect" inputs from E3SM data, then get joined & fine-tuned together. A key design choice: a stochastic atmosphere. Its randomness becomes the coupled system's own natural variability.
Against 400 years of E3SM data the emulator never trained on, its average climate holds up: temperature & precipitation biases are smaller than E3SM's own gap from real-world observations. El Niño (ENSO) stays realistic across its 400-year run without drifting or collapsing.
One important limitation: the emulator captures daily rainfall up to the 99.99th percentile but underestimates the very rarest tropical downpours. Learning the shape of a distribution is not the same as reaching its far tail.
Our coupled emulator could help screen ideas before running E3SM itself & enable large ensembles that cheaply sample climate variability. Next, we’ll explore historical runs where greenhouse gases + other climate drivers change over time.
📝 Learn more: https://e3sm.org/a-fully-coupled-ai-emulator-of-e3smv3-reproduces-its-statistics/
🤗 Download: https://huggingface.co/allenai/SamudrACE-E3SMv3
Do LLM safety & capability evals measure what they claim to? We built BenchMIRT to audit benchmarks and identify which model abilities their questions actually test. 👇
BenchMIRT builds on Item Response Theory (IRT), a psychometrics technique for measuring abilities from patterns of test responses. The key idea: not every question is equally informative. Some are harder, while others better distinguish stronger models from weaker ones.
We trained BenchMIRT on results from 100 LLMs across 16 benchmarks and 34K+ questions. Without telling it what the benchmarks were designed to measure, two dominant dimensions emerged: general reasoning and safety.
BenchMIRT can estimate:
◙ A model’s strength on the capabilities reflected in the benchmark set
◙ A question’s difficulty and how strongly it distinguishes models along those capabilities
That lets us audit what popular evals are really measuring.
For example, BenchMIRT finds that HarmBench mostly tests safety behavior—but its copyright questions depend more on reasoning. XSTest draws on both reasoning and safety, while ToxiGen provides little signal on either.
BenchMIRT can also make evals more efficient. Keeping just the strongest 10% of questions preserves nearly the same picture of model strengths as the full benchmark set. And on held-out questions, BenchMIRT predicts whether a model will answer correctly 79% of the time, versus 70% for a simpler benchmark-average baseline.
BenchMIRT gives researchers a clearer view of what benchmarks actually measure—and could help make evals smaller, more focused, and easier to interpret.
We’re releasing it openly:
📖 Learn more: https://allenai.org/blog/benchmirt
💻 Code: https://github.com/allenai/BenchMIRT
📊 Dataset: https://huggingface.co/collections/allenai/benchmirt
📄 Paper: https://allenai.org/papers/benchmirt
At an event on August 27, we brought together AI researchers, scientists, & medical practitioners to explore what AI needs to do better to meaningfully advance science. Five ideas kept coming up:
1. AI still needs human scientific judgment.
A system can surface a statistically surprising result. That doesn’t mean it’s biologically plausible, important, or worth pursuing. Scientists still need to decide which findings matter—and why.
2. Scientific AI needs to be steerable.
Research rarely follows a fixed path. New evidence comes in. Hypotheses change. Researchers bring in new datasets or tools. AI systems need to adapt as the research evolves, without forcing scientists to start over**.**
3. Some scientific tasks are easier to hand off to AI than others.
AI can handle well-defined work like literature search, where results are easy to check. Proposing new mechanisms or experiments is harder; those ideas still need testing.
4. AI can amplify bad science, too.
More AI-driven analyses won’t fix weak data, flawed study design, or bad assumptions. As AI becomes more powerful, the fundamentals of good science become more important—not less.
5. One promising direction is a tighter loop between AI and experiments.
AI could synthesize evidence, help decide what to test next, & use the results to shape the next question. The goal isn’t an AI scientist working alone, but a system scientists can keep guiding.
These ideas came out of presentations + a panel with Bodhisattwa Prasad Majumder (Ai2), Abraham Flaxman (University of Washington & IHME), Hoifung Poon (Recursion), Sasha Stanton and Kelly Paulson (Providence), Kyle Travaglini (Allen Institute), and Stephen Salerno (WashU).
Thanks to everyone who joined us.
→ Learn more: https://allenai.org/blog/swedish-autodiscovery-recap
AI’s biggest role in science may not be answering questions. It may be helping scientists identify which questions are worth asking.
That’s the idea behind AutoDiscovery, our system for open-ended scientific exploration.
Providence researchers at the Paul G. Allen Research Center and the Earle A. Chiles Research Institute put it to work on The Cancer Genome Atlas, one of the world’s most studied cancer datasets. AutoDiscovery generated and tested hypotheses across the data, looking for patterns that challenged expectations.
One involved invasive lobular carcinoma (ILC), a breast cancer subtype affecting roughly 48,000 Americans each year. ILC has long been considered “immune cold”—but AutoDiscovery surfaced more immune activity than expected.
Providence Swedish researchers checked a separate patient dataset and found the same pattern. They then analyzed ILC tumor samples in the lab and found T-cells around the tumors.
Together, the evidence suggests ILC warrants broader investigation for immunotherapy.
This points toward the kind of relationship between scientists and AI that we’re interested in building: systems that can explore more possibilities than a person could reasonably test one by one, paired with human expertise to determine which results matter and how to investigate them rigorously.
Providence Swedish is now expanding its use of AutoDiscovery to additional protected cancer data.
🔗 Learn more: https://allenai.org/blog/swedish-autodiscovery-partnership📄 Read the scientific report: https://allenai.org/papers/autodiscovery-swedish-cancer
A Thai research team adapted our Dolma data-curation toolkit to build Mangosteen, a 47B-token corpus for Thai LLMs. They used Dolma to filter widely used web datasets into a smaller corpus that improved Thai LLM performance despite using less data.
The team saw a problem with existing Thai pretraining data: it often leaned heavily on web crawls, hadn’t been extensively audited by Thai speakers, and missed useful sources like books, research papers, authoritative sites, & YouTube subtitles.
Dolma gave the team a strong starting point, so they didn’t have to build a data-curation pipeline from scratch. Because it’s open, they could keep what worked and change what didn’t for Thai—including deduplication, quality filters, & language-specific tools.
That work became Mangosteen, the team’s 47B-token Thai pretraining corpus. Models trained on it matched or beat Thai LLMs trained on larger web datasets after filtering out lower-quality and duplicate text—showing careful curation can mean less data without worse models.
→ Learn more in our new blog: https://allenai.org/blog/thai-llm-dolma
A Georgia Tech team used our fully open Olmo stack to trace performance on social and general reasoning, plus social-science and STEM knowledge tests, back to the kinds of text the model trained on.
Using millions of files from Olmo 3’s public training corpus, Dolma 3, and influence functions that estimate how much individual files affected a model’s answers, they found distinct patterns. For example, dialogue-rich, interpersonal writing was more influential for reasoning than for factual knowledge questions.
This kind of analysis depends on access to more than model weights. By opening Olmo’s training data and scientific stack, we make it possible for independent researchers to investigate where model capabilities come from—and test those connections directly.
Learn more in our new blog: https://allenai.org/blog/olmo-capability-tracing
Researchers at UT Austin, Northeastern, and MD Anderson used our fully open Olmo 3 to investigate whether LLMs actually know specific drugs—or infer from patterns in their names.
For 51–59% of tested drugs, Olmo 3 showed little evidence of drug-specific knowledge. Another 12–18% appeared driven by affixes like “-pril” or “-olol,” which can reveal a drug’s class.
Because we release model weights, training data, documentation, and intermediate checkpoints, the researchers could trace the behavior further. Using our infini-gram engine for searching massive text corpora, they found that drugs appearing less often in training were more likely to trigger these naming shortcuts.
It’s a useful example of what fully open models enable: not just spotting a model behavior, but investigating where it comes from.
Today we're introducing a preview of TutorMoments, a framework that measures whether AI tutors can make one of the hardest calls in teaching: when to step in and help a student, & when to hold back and let them do the heavy thinking. 👇
Language models are trained to be helpful, and a helpful assistant tends to do the hard part of learning for you: explains the concept, lays out the steps, & guides you to the answer. That can cut short the productive struggle that leads to stronger understanding.
TutorMoments is built on transcripts of real one-on-one math tutoring. We had experienced teachers read them & flag key moments—decision points where the tutor had to choose between making a problem easier & pushing the student to do more of the reasoning.
TutorMoments pauses a transcript at these key moments & lets an LLM take over as the tutor; another model stands in for the student. Each replay is scored: did the tutor support the student when needed, push for harder thinking when they were ready, & avoid over-helping?
In our replays, models told only to "tutor well" tend to over-help, providing lots of support but rarely pushing toward deeper thinking. That suggests a model's default helpful-assistant behavior isn't enough on its own to tutor well.
Spelling out the trade-off in the prompt helps—every model we tested scores higher once told when to help vs. when to hold back. But it only goes so far. Models still differ widely in how reliably they make that call, & even the best scorers have plenty of room to improve.
To gather feedback, we're releasing de-identified annotated tutoring transcripts plus code & scored replays.
💻 Code: https://github.com/allenai/tutormoments
🤗 Data: https://huggingface.co/datasets/allenai/tutormoments-preview
🌐 Learn more: https://allenai.org/blog/tutormoments
We're expanding our partnership with Hugging Face to accelerate open science. 👇
Our storage on the Hub is roughly tripling to ~2 petabytes, & our downloads now run at high speed—even for our largest datasets & multi-checkpoint models.
The added capacity reflects our footprint on the Hub – 900+ models & 1,200+ datasets – with room to grow. According to Hugging Face's official heatmap, we publish more new artifacts each year than any other organization it tracks: https://huggingface.co/spaces/cfahlgren1/model-release-heatmap
This builds on recent work together:
⦿ Hugging Face made olmOCR-Bench an official Hub leaderboard, making it easier for the community to evaluate and compare document understanding models on a shared benchmark.
⦿ When Hugging Face brought MolmoAct 2 to LeRobot, we released its training data in LeRobot’s format—ready for researchers and developers to use on real hardware.
We look forward to working with Hugging Face to deliver new artifacts in the coming months, including multimodal models & open models for science.
📝 Learn more in our blog: https://allenai.org/blog/hugging-face-partnership
🤗 Browse our models, datasets, & benchmarks: https://huggingface.co/allenai
📸 140+ people joined us during #SeattleTechWeek to learn how AI gets built at Ai2.
Our Olmo research lead, Iz Beltagy, walked through what it takes to train a fully open language model—from large-scale training runs to fine-tuning, reinforcement learning, post-training, and evaluation before release. Thanks to everyone who came, asked thoughtful questions, and stayed afterward to continue the conversation.
Want to help us build the next generation of fully open AI? We’re hiring across research, engineering, product, operations, & more. See our open roles: https://allenai.org/careers