The Distillation Storm: The AI Technology Race Companies Don’t Want to Discuss Publicly
Everyone is unwilling to talk about it publicly, yet everyone is quietly watching it.
Some believe it is a disreputable form of theft. Others believe it is merely an optimization technique that has been stigmatized by a handful of leaders for their own interests.
Over the past several months, many threads in the AI field have converged on the same node—distillation.
Changes and events associated with it include: open-source models closing in on the strongest closed-source models; 77 U.S. companies signing an open letter opposing hasty restrictions on open-source models; Anthropic twice accusing Chinese companies of using large numbers of fraudulent accounts to extract data; a $1.5 billion copyright-infringement settlement; Zhang Yiming’s direct response about “not distilling” at a ByteDance Seed all-hands meeting…
This technology, which has existed for many years, has repeatedly been mentioned and discussed in 2026, while also being misunderstood and distorted. What exactly is distillation? How is large-scale distillation carried out? Can distillation become a moat for a model-development team? And what is its cost?
We interviewed nearly ten model researchers and practitioners from different companies, and combined their accounts with public research and technical reports to reconstruct the past and present of distillation, as well as the broader changes it is now bringing about.
The origin of distillation: for compression, not for becoming stronger
Distillation is not plagiarism. It is not theft of software code, nor can it directly obtain another model’s weights or complete training data.
The kind of distillation that is currently controversial—that is, distillation intended to make a model stronger—is technically a way of obtaining high-quality data: repeatedly ask questions of a stronger “teacher model,” obtain its answers, and then use these “question-answer” data pairs to train another “student model,” allowing the latter to achieve similar performance.
The idea of distillation has existed for a long time. In 2015, Geoffrey Hinton, who had recently joined Google Brain, published Distilling the Knowledge in a Neural Network together with then-Google Brain head Jeff Dean and young researcher Oriol Vinyals. For the first time, they summarized earlier ideas such as Model Compression (2006) under the concept of “distillation.”
The idea of distillation has existed for a long time. In 2015, Geoffrey Hinton, who had recently joined Google Brain, published Distilling the Knowledge in a Neural Network together with then-Google Brain head Jeff Dean and young researcher Oriol Vinyals. For the first time, they summarized earlier ideas such as Model Compression (2006) under the concept of “distillation.”
At the time, Google’s proposal of the Transformer—the architectural foundation of today’s large language models—was still nearly two and a half years away. Hinton and his colleagues applied the idea of distillation to image-recognition models: the method was to have the student model learn the probability distribution output by the teacher model.
For example, recognizing that a cat is a cat is a statistical process for deep learning: 0.7 cat, 0.2 fox, 0.1 dog → cat.
The student model can see this probability distribution output by the teacher. This is learning the “logits,” in what is known as “soft distillation.” (Logits are a set of raw scores that become a probability distribution after being transformed by Softmax.)
Soft distillation is usually “white-box distillation,” because it requires the teacher model’s output probabilities to be fully accessible to the student model.
This kind of distillation usually happens within the same organization. Its purpose is not to make the model stronger, but to “compress” it—to use a model with fewer parameters to approximate the capabilities of a model with more parameters. This sacrifices some performance, but makes inference faster and cheaper.
Even today, “compression” remains one of the most typical uses of distillation.
For example, in autonomous driving, companies may first build a more powerful large model in the cloud, then use methods such as distillation and pruning to turn it into a smaller model that can run on an in-vehicle chip. Companies including Li Auto and XPeng have carried out such practices.
A more recent example is DeepSeek-R1 in early 2025. At the time, DeepSeek simultaneously released six smaller distilled models, all of which used R1 itself—a model with 671 billion total parameters—as the teacher model. Among the student models, four were based on Alibaba’s Qwen2.5 and two on Meta’s Llama 3. The smallest had 1.5 billion parameters and the largest 70 billion.
DeepSeek first had R1 generate roughly 600,000 reasoning data samples in the form of “question-reasoning process-answer,” as well as roughly 200,000 non-reasoning data samples. It then used these data during post-training to perform supervised fine-tuning (SFT) on six smaller base models. All of these models acquired stronger reasoning capabilities.
LatePost previously reported that after the 2026 Chinese New Year holiday, Guo Daya, one of the core authors of R1, joined ByteDance Seed.
“Industrial-scale distillation attacks”
When the purpose is compression, distillation is a neutral technical method. But by February 2026, Google and Anthropic had both published articles pairing the words “distillation” and “attack,” directly accusing some companies of using distillation for unfair competition.
Google considers this a form of IP theft:
>
Anthropic said separately in February and June that DeepSeek, Moonshot AI (Kimi), MiniMax, and Alibaba Qwen had collectively used roughly 50,000 fraudulent accounts to conduct more than 44.8 million interactions with Claude in an attempt to extract Claude’s capabilities.
(February: Detecting and Preventing Distillation Attacks; June: Anthropic’s letter to the U.S. Senate.) The companies mentioned did not issue direct responses.
Anthropic said separately in February and June that DeepSeek, Moonshot AI (Kimi), MiniMax, and Alibaba Qwen had collectively used roughly 50,000 fraudulent accounts to conduct more than 44.8 million interactions with Claude in an attempt to extract Claude’s capabilities.
(February: Detecting and Preventing Distillation Attacks; June: Anthropic’s letter to the U.S. Senate.) The companies mentioned did not issue direct responses.
How can distillation of leading closed-source models be carried out at large scale? The source of the change is still technological. Three threads can be identified:
From soft distillation to hard distillation, and from white-box distillation to black-box distillation
In 2016, Yoon Kim, then a PhD student at Harvard, and his adviser Sasha Rush proposed sequence-level knowledge distillation, applying a method originally used in image recognition to the language task of translation.
Sequence-level distillation no longer learns the probability distribution of the model’s output at every step. Instead, it first has the teacher model generate high-quality translations, and then has the student model learn from complete “source text-translation” sequence pairs.
The technique was originally proposed to compress enormous translation models and improve decoding speed, but it also had another effect: distillation no longer required knowing the teacher model’s step-by-step output probability distribution. Distillation could be achieved simply by looking at the “final answer.”
This is “hard distillation.” It can be conducted in a black-box setting: one only needs to call an API and directly obtain the teacher model’s answers.
Sasha Rush later joined AI coding company Cursor in March 2025. In June this year, Cursor was acquired for $60 billion by SpaceXAI—the new name following the merger of SpaceX and xAI—and its team was incorporated into the company.
Sasha Rush joined AI coding company Cursor in March 2025. In June this year, Cursor was acquired for $60 billion by SpaceXAI—the new name following the merger of SpaceX and xAI—and its team was incorporated into the company.
Image
The rise of reasoning models
In September 2024, just as discussion was spreading about Scaling Laws “hitting a wall,” OpenAI released the o1 reasoning model. o1 brought two changes:
- Large-scale reinforcement learning (RL) during post-training can teach models to develop reasoning strategies;
- During model inference—that is, when the model is being used—more test-time compute can be used to allow the model to generate longer chains of thought when answering complex questions, enabling continued performance improvements.
Both changes can amplify the effectiveness of distillation.
First, large-scale distillation is a method mainly used during post-training. As post-training becomes more important, the return on investment from distillation also increases.
At the same time, test-time compute means that, in addition to producing a final answer, a model also produces long reasoning processes such as chains of thought and reasoning trajectories—including chains of thought, tool calls, search processes, error-correction processes, and so on. These outputs can serve as raw material for distillation data.
Four months later, the distillation process disclosed in DeepSeek’s R1 technical report provided a more detailed demonstration of how distillation can make a model stronger. R1 included several specific practices and findings.
First, the distillation process primarily used “question-reasoning process-answer” data generated by R1. This is currently the ideal form of distillation and works better than data pairs containing only “question-answer.”
When releasing R1, the relatively open DeepSeek directly displayed complete chains of thought, saying it “hoped this would help the community distill better small models.”
By contrast, leading closed-source model companies—OpenAI, Anthropic, and Google DeepMind—have consistently hidden complete chains of thought and reasoning trajectories from users.
This is why practitioners, when talking about distillation, often say things like: “So-and-so cracked so-and-so’s chain of thought.”
This Monday, August 10, researchers from the University of Tübingen in Germany and other institutions released a paper titled Stealing Reasoning Traces from Proprietary LLM APIs, demonstrating several methods they discovered for reconstructing reasoning trajectories.
Image
The study’s website: stolen-thoughts.com
The study’s website: stolen-thoughts.com
In fact, this has long ceased to be a secret—even when closed-source model companies deliberately hide them, chains of thought and reasoning trajectories can still be reconstructed through technical means.
Fundamentally, this is because they are all part of the model’s output. They are products of the model-use stage. As long as you use a model, chains of thought and reasoning processes will be generated, and they will leave traces. This still falls under black-box distillation and hard distillation.
Another finding from R1 about distillation was that, during post-training, direct distillation brought greater improvements than having the model perform reinforcement learning on its own.
DeepSeek conducted a controlled experiment at the time using Qwen2.5-32B:
DeepSeek conducted a controlled experiment at the time using Qwen2.5-32B:
- Performing more than 10,000 steps of large-scale reinforcement learning on Qwen-32B-Base produced a model that scored 47.0% on AIME 2024, which evaluates mathematical problem-solving ability;
- Supervised fine-tuning of Qwen-32B-Base using 800,000 samples generated by R1 produced a model that scored 72.6% on AIME 2024, more than 25 percentage points higher than the RL method.
>
Although this was the result of one specific experiment and cannot necessarily be generalized to all situations, it remains a highly attractive finding.
Long-horizon reinforcement-learning training is inherently more difficult than supervised fine-tuning. It places greater demands on infrastructure, is often slower, and requires more compute.
DeepSeek publicly demonstrated a relatively economical, efficient, and reliable way to improve the reasoning capabilities of smaller or weaker models.
Multiple practitioners said that some recent practices involve relying primarily on supervised fine-tuning during post-training, while doing almost no reinforcement learning—or very little—and still achieving very good results.
It was also throughout the period following DeepSeek-R1, from 2025 to the present, that companies including Anthropic, OpenAI, and Google said the “distillation attack” behavior they detected had continued to increase.
During the same period, further exploration of post-training also popularized another, noncontroversial distillation practice: on-policy distillation, currently used primarily for integrating capabilities during post-training.
The difference between on-policy and off-policy distillation lies in who generates the data.
Most of the aforementioned distillation in which one model learns from another closed-source model is off-policy distillation: the teacher model generates the data.
In on-policy distillation, by contrast, the student model generates the reasoning and answers, while the teacher model provides feedback. The feedback may consist of token-by-token output probability distributions, in a white-box setting, or judgments on reasoning trajectories and answers, in a black-box setting.
Image
From the second half of 2025 to the present, Alibaba Qwen, Thinking Machines Lab, and Xiaomi MiMo have all contributed practices and improvements related to on-policy distillation.
From the second half of 2025 to the present, Alibaba Qwen, Thinking Machines Lab, and Xiaomi MiMo have all contributed practices and improvements related to on-policy distillation.
At the beginning of this year, Xiaomi introduced the MOPD—Multi-Teacher On-Policy Distillation—method in the MiMo V2-Flash technical report. In June, it published a separate paper, Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training.
The method first separately trains teacher models specializing in mathematics, coding, tool use, and other areas, then lets the student model generate its own trajectories and receive feedback from different teachers depending on the task.
It is intended to solve a new problem in post-training: if capabilities from multiple domains are directly mixed together for reinforcement learning, they often interfere with one another, causing gains in one area to come at the expense of another.
The technical reports for DeepSeek-V4 and Kimi K3 both said that MOPD-style approaches were used during post-training to combine multiple expert models.
Using AI to accelerate AI
Using AI to accelerate AI
The third thread behind the scaling of distillation is that distillation itself is becoming increasingly automated as AI capabilities grow stronger.
Return to the key element of distillation: “question-reasoning process-answer” data pairs.
First, the question-asking stage can shift from humans to AI. Since 2022, numerous studies such as Self-Instruct have attempted to solve the problem that high-quality questions are scarce and expensive.
In practice, companies can first select high-quality real questions from authorized user behavior, then use those real questions as a basis for expanding into additional AI-generated questions, which are asked of the teacher model at much higher frequency to obtain more answers.
It is like starting with some yeast—real data—and using it to ferment a much larger batch of dough.
Entire “question-reasoning process-answer” data pairs can likewise be rewritten and expanded in this way.
Fundamentally, this reflects a basic mode of thinking in today’s AI field: using AI and models to automate and accelerate AI itself.
Many specific steps throughout the distillation process can also make use of AI—for example, selecting high-quality questions from enormous volumes of real queries, evaluating “what constitutes high-quality data,” and increasing the diversity of synthetic data.
Some stages can be handled by models. In others, increasingly powerful AI coding capabilities can be used to build and improve various systems faster, accelerating and optimizing the overall process.
It can be said that from 2025 to the present, the methods and practical experience required for large-scale distillation have gradually matured. Its effectiveness and necessity have also become increasingly evident as the importance of post-training has risen.
Discussion surrounding “distillation” has gradually moved beyond the AI technical community. In the process of spreading, it has been simplified, misunderstood, and even distorted.
Distillation is no longer a purely technical issue. It has moved into the center of the storm.
Misunderstandings about distillation: it is neither a silver bullet nor a secret
Having reviewed the general principles of distillation, we now have a better foundation for discussing the various debates surrounding it.
· Distillation is not a silver bullet and is not the most important factor determining model performance
As repeatedly mentioned above, distillation that makes large language models stronger mainly occurs during post-training, though it can also occur during mid-training.
The overall performance of a model, however, comes from the complete training process from pre-training through post-training. Pre-training is generally considered more important.
Therefore, distillation is not a silver bullet and is not the most important factor determining a model’s performance.
In DeepSeek-R1’s technical report, both models were distilled using 800,000 R1-generated data samples. After distillation, the Qwen2.5-32B base model scored 72.6% on AIME 2024, higher than the 70.0% scored by distilled Llama-3.3-70B-Instruct—even though the latter had more than twice as many parameters.
After K3 was released, Ai2 researcher Nathan Lambert replied to a tweet about K3 reaching No. 1 on the Frontend Code Arena leaderboard, which evaluates frontend coding ability, saying: by this point, the whole “distillation” narrative should stop; people should acknowledge that China is also extremely good at building models.
Reaching a certain level of pre-training capability is a prerequisite for Chinese open-source models such as K3, GLM-5.2, and DeepSeek-V4 to achieve their current performance.
· Distillation requires substantial operations, experience, and engineering know-how
Distillation is often compared to a shortcut, and shortcuts usually imply something easy and effortless.
In reality, however, conducting large-scale distillation today is a fairly complex systems-engineering undertaking.
Based on the descriptions of multiple practitioners, there are several particularly difficult aspects of large-scale distillation.
The first is being able to call leading models stably, frequently, and at large scale, while also carrying out user operations.
One practice that was mentioned is this: build a large number of intermediary services and attract a group of specific users with genuine usage behavior through discounts or other means. These users might be senior programmers, or science and engineering students and researchers who need to handle large volumes of research questions.
In their everyday use, they naturally generate high-quality, multi-turn queries arising from real scenarios and real tasks, and then obtain answers from the model.
After these questions and answers are selected and processed in certain ways, the resulting data can serve as the starting point for fermenting and expanding still more data.
This tests the team’s operational capabilities—whether it knows where high-quality users are and how to reach them—as well as its engineering capabilities in building the system, such as whether the system can remain sufficiently stable.
It also requires certain ecosystem capabilities, because the process may require collaboration with third-party companies or institutions.
The second challenge is the team’s own ability to construct high-quality questions and tasks.
This requires a deep understanding of the tasks, of the data, and of the performance boundaries of current leading models. These capabilities overlap with some of those required for training large models themselves.
The third is how to make good use of the data.
This requires building a data pipeline: whether the task distribution is reasonable; how to sample, select, filter, deduplicate, expand, and correct the data; and how to determine formats and ratios.
There are measurable optimization indicators for such a pipeline. For example: of the raw data obtained, what proportion ultimately gets used in post-training? What are the efficiency and quality of data expansion?
The quality of the data pipeline affects not only performance, but also efficiency and cost.
Continuous distillation is also extremely expensive.
There are various claims circulating about how much companies in the industry are spending and budgeting for distillation this year, ranging from more than $100 million to as much as $1 billion.
An AI investor said: distillation is not a simple button. It is not something where you press it once and model performance suddenly skyrockets. There are still many implementation problems involved. Distillation also requires calculating return on investment.
· Is distillation a moat for a model team?
So, can large-scale distillation, which is relatively complex to implement, become a moat for a model-development team?
Most practitioners we spoke with—whether they came from companies rumored to be doing distillation or from companies that do not distill—had relatively similar views: for first-tier companies, distillation does not constitute a long-term moat.
Like many technologies in large models, ideas and practices around distillation gradually spread.
Personnel movement, open-source sharing, conference exchanges, third-party service providers looking for more customers… People and information in the AI world are constantly moving around.
We repeatedly heard researchers make essentially the same observation: there are no real secrets in large models.
Most of the competitive advantage created by a technical method itself is first-mover advantage. Those who start earlier accumulate more experience, but this is not the kind of insurmountable, winner-takes-all moat created by network effects.
In the large-model field, the phenomenon that is genuinely considered capable of creating a strong competitive moat is the “data flywheel.”
If one company’s model is strong enough to reach a large number of users who use it for highly difficult tasks, it will receive more high-quality data flowing back from those users.
Moreover, those data are unique, nonpublic, and unavailable to others.
After appropriate processing, those data can be used to help train an even stronger model, which in turn attracts more users to use it for even harder tasks.
Within this flywheel logic, applications that directly interact with users are extremely valuable.
For example, depending on user agreements and permission settings, applications such as Cursor, Devin, and Manus may potentially obtain more complete data and user behavior than the models they call.
Yesterday, August 12, after Grok 4.6 was released, Musk replied to a tweet announcing that Devin had integrated Grok 4.6:
>
The $60 billion acquisition of Cursor appears to have been well worth it.
Image
However, the data flywheel also has controversies of its own.
Model and application companies can obtain data, but are they allowed to use those data for training?
For harder tasks and higher-value scenarios, will customers and users increasingly prefer to retain control over such data themselves?
At the same time, as a model’s number of users continues to grow and user types and scenarios become increasingly diverse, is it still worth sifting through all that sand for a few grains of gold?
Some practitioners believe that in certain AI applications focused on lifestyle assistance and entertainment, the overwhelming majority of user-generated data is garbage when it comes to training stronger models.
The internal line of judgment: black box or white box
One of the most interesting phenomena surrounding distillation is this:
No one wants to discuss it publicly, yet most of the practitioners we spoke with do not, deep down, consider it an extremely shameful practice that violates their technical beliefs.
The dividing line in their internal judgment lies between black box and white box.
At present, distillation performed on closed-source models is black-box distillation.
The data it obtains are all data generated during use after those models have been released. They are products of the model as a product.
So why should other companies not be allowed, as users, to ask that model questions and obtain answers?
After all, everyone has paid real money for those questions and answers.
(Of course, in practice, people use various methods to exploit discounts, promotions, or loopholes and drive the cost down.)
The more controversial part here is reasoning trajectories.
Most models hide their complete reasoning trajectories, so the party conducting distillation needs to use certain technical methods to infer and reconstruct them.
But reasoning trajectories are still products of the model-use stage.
Anthropic, Google, and OpenAI would say: our user agreements explicitly stipulate that competitors may not use our models for the purpose of training and improving their own models.
But who was sued by The New York Times for allegedly copying and using, without permission, the newspaper’s archive of journalism accumulated over more than 170 years by generations of reporters, commentators, and writers?
OpenAI.
Who downloaded enormous numbers of books from pirate platforms, unwilling to pay for any of them, was collectively sued by several U.S. authors, and has just reached a $1.5 billion settlement?
Anthropic.
Even some American AI practitioners believe companies such as Anthropic are highly hypocritical on this issue.
It is somewhat like the situation in Christopher Nolan’s new film The Odyssey: you sent the wooden horse into Troy, and now your own homeland is being attacked by people arriving from across the sea.
Large-scale distillation is a new issue that emerged following the development of new technologies.
It is difficult for most people to be sincerely convinced that it is unacceptable, unethical, or even shameful and evil merely because a handful of companies say it violates their user agreements.
Moreover, violating a user agreement is not equivalent to committing legal infringement. Other laws, jurisdictional issues, and so on are also involved.
Distillation in the broadest sense—that is, using data to improve models—has already become ubiquitous.
One widely circulated method of “verifying” distillation is actually invalid:
If you ask a model, “Who are you?” and Model A says it is Model B, that is not ironclad proof that A distilled B.
During pre-training, everyone uses vast amounts of public internet data, and much of that data already consists of content generated by the models themselves.
In July this year, Nvidia founder Jensen Huang was asked about distillation in an Axios interview. He said:
>
For most model companies, including some American model companies, the accelerator on distillation has already been pressed down.
Few will voluntarily step off it in the short term.
Why Zhang Yiming chose not to distill
Most companies are unwilling to discuss distillation publicly. ByteDance is an exception.
At the most recent Seed all-hands meeting, held roughly half a month ago in late July, ByteDance founder Zhang Yiming explicitly said: he opposes distillation.
ByteDance has gone through adjustments in its approach to distillation.
At the end of 2023, ByteDance was the first major Chinese large-model company to be publicly identified by OpenAI as being suspected of improperly using outputs from GPT models to improve its own model.
At the time, large-scale distillation had not yet emerged.
OpenAI itself also said that ByteDance’s API usage was minimal.
ByteDance later responded that GPT-generated data had previously been used for model annotation and evaluation, but that the relevant data had been removed from its training set in mid-2023.
According to LatePost, during the more than two years that followed—including the entire year of 2025, when the scale of distillation expanded rapidly—ByteDance Seed did not distill leading closed-source models.
Instead, it obtained data through other means, such as hiring medalists and prize winners from mathematics, computer science, and other science-and-engineering competitions at high salaries to construct and annotate data.
During the same period, the controversy surrounding TikTok’s U.S. business remained unresolved.
It was not until the end of January 2026 that the restructuring transaction for TikTok’s U.S. business was formally completed.
Around the 2026 Chinese New Year holiday, Seed briefly went through a period of wavering.
At the time, OpenClaw was becoming hugely popular, Anthropic’s revenue was surging, usage of Chinese open-source models such as GLM-5 was rising sharply, and quantitative improvements in coding ability had begun to produce qualitative changes.
ByteDance’s models, meanwhile, were relatively behind in coding ability.
After a period of deliberation, judging from Zhang Yiming’s remarks at the Seed all-hands meeting, ByteDance has made its choice:
It will not distill external leading models.
According to an exclusive report by LatePost, Zhang Yiming’s views at the all-hands meeting were:
- Distillation can improve model performance in the short term, but fundamentally it is still copying capabilities that Claude already possesses. Following this path, at best you can keep getting closer to the other side; it is difficult to genuinely surpass it.
- He also hopes Seed can build its own moat in AGI at a more fundamental level.
- Even if refusing to distill means the company will temporarily fall behind domestic competitors technologically, it will not use this shortcut to advance ByteDance’s model capabilities.
This leads to another question about distillation:
Can distillation really not surpass the teacher model?
We asked multiple practitioners this question as well. Their answers were relatively similar:
Technically, it is not impossible, but there are potential organizational risks.
Distillation has certain inherent technical problems. For example, it may cause the student model to learn some of the teacher model’s errors, biases, refusal habits, and expressive patterns.
But distillation is only one stage of model training.
There are many other parts of the complete model-training process that can be improved: pre-training data, architecture, algorithms, infrastructure…
If multiple optimizations accumulate, is it possible for a student model to become better than a particular teacher model?
Some studies have already shown that on certain specific tasks, student models can surpass their teacher models.
For example, in December 2024, Microsoft released the 14-billion-parameter Phi-4. A large portion of its training data consisted of data synthesized by teacher models including GPT-4o.
Phi-4 surpassed GPT-4o on two benchmarks:
- GPQA (graduate- and PhD-level scientific knowledge and reasoning): Phi-4 scored 56.1%, versus GPT-4o’s 50.6%;
- MATH (mathematical problem-solving and reasoning): Phi-4 scored 80.4%, versus GPT-4o’s 74.6%.
One unusual aspect was that, as a relatively small-parameter model, Phi-4 directly used teacher-generated data during pre-training.
But when genuinely training ultra-large-scale models on several trillion tokens of data, it is difficult to make much use of data obtained through distillation during the enormous pre-training stage.
Compared with directly processing all kinds of web pages, code, and books, calling a teacher model one sample at a time to generate data is slow and expensive.
However, this also raises a possibility:
As model inference becomes dramatically faster and prices fall dramatically, could distillation—or, more broadly, data constructed with the help of stronger models—be used more extensively during pre-training as well?
At the same time, could methods such as multi-teacher distillation also be used to make models stronger?
In theory, one student model could simultaneously learn from multiple strongest-in-class models such as Claude and GPT.
This could introduce new technical problems. For example, distilling from different base models may lead to training instability and interference between different capabilities.
Still more aggressive speculation includes this question:
Could the very leading companies distill their own models into themselves, achieving a kind of “left foot stepping on the right foot” self-propelled ascent?
From a purely research perspective, whether “a model that uses distillation can surpass its teacher model” is at the very least a question that remains to be tested and explored.
Zhang Yiming and ByteDance, meanwhile, have given their own answer through their actions.
He believes it cannot.
This may be related to the organizational risks mentioned by multiple practitioners.
Distillation is a relatively economical method that produces results quickly.
An athlete can, of course, both take performance-enhancing drugs and train diligently.
But in reality, it is often difficult to do both, because people become opportunistic and complacent.
Once a team devotes a relatively large share of its attention and resources during a certain period to distillation, projects and individuals exploring more uncertain, longer-term directions may not receive sufficient resources or recognition.
ByteDance is making a bet:
Building data capabilities that do not depend on external competitors can create a more robust and long-term technological advantage in models.
Starting this June, according to Intelligent Emergence, ByteDance began reorganizing its data teams, creating a first-level AI division called “AI Data and Security”, on the same organizational level as Seed and Flow.
As mentioned above, most people do not believe the methodology of distillation constitutes a long-term moat.
What genuinely has the potential to create a strong moat in the large-model field is the data flywheel.
Data itself has also become a high-value standalone part of the industry.
For example, the U.S. company Mercor’s main business is helping large-model companies find scientists, PhDs in various disciplines, and other professionals to carry out data construction, annotation, and model-evaluation tasks.
According to reports, in July this year Mercor was seeking a new funding round at a valuation of $20 billion.
In China, there are also companies that started with data construction whose valuations have already reached $3 billion.
Under our exclusive report about “ByteDance not doing distillation,” there was one comment:
>
No one can easily hit the brakes
When someone accuses you of doing something wrong, responding with “Didn’t you do the same thing before?” does not resolve the conflict.
At this very moment, open-source models are moving ever closer in performance to the most advanced closed-source models.
And the objective reality is that the strongest open-source models come from China, while the strongest closed-source models come from the United States.
Previously, Zhipu AI had already been placed on the U.S. Department of Commerce’s Entity List in January 2025.
The Intelligence Authorization Act for Fiscal Year 2026, which took effect at the end of that year, requires DeepSeek to be removed from U.S. intelligence systems, national-security systems, and relevant suppliers.
By April this year, the U.S. House Committee on Homeland Security and the House Select Committee on Strategic Competition Between the United States and the Chinese Communist Party had begun investigating American companies’ use of Chinese models.
They asked Cursor why it wanted to use Kimi K2.5 as the base model for Composer.
They asked Airbnb why it wanted to use Alibaba Qwen in its customer-service operations.
After Kimi K3 was released on July 16, reports said that the U.S. government was considering restricting or even banning certain Chinese open-source models.
On July 24, Microsoft, Nvidia, Meta, Fireworks AI, and a total of 77 other companies and organizations successively signed an open letter titled Open Weights & American AI Leadership, opposing hasty restrictions on open-source models.
Jensen Huang’s first-ever tweet on Twitter was sharing this open letter.
The new restrictive measures discussed in recent months are still under consideration and have not yet been implemented.
Image
Companies including Anthropic, OpenAI, and Google are also adopting stricter technical measures to identify and ban accounts suspected of being used for distillation.
For example, Anthropic says it has established classifiers and behavioral-fingerprinting systems for identifying distillation traffic.
These systems can detect coordination across accounts, repeated questioning, and attempts to extract chains of thought.
At the same time, it will strengthen identity verification for education, research, and startup accounts.
In the University of Tübingen “stealing reasoning traces” study mentioned earlier, the methods the researchers used to reconstruct reasoning trajectories were reported by the research team to the relevant closed-source model companies.
By the time the researchers uploaded their paper, some of those methods had already stopped working.
All of the parties involved are making their own choices and preparations.
The changes that follow will reshape the entire industry chain, from compute and cloud services to infrastructure, models, applications, and customer deployment.
The storm is still continuing.
The next eye of the storm will not necessarily be distillation. Distillation itself is only one of many methods for optimizing models.
Why are so many companies investing so many resources and rushing headlong into this race for intelligence?
In the first half of this year, explosive growth in coding and agents reversed market expectations.
Going forward, if the scale and speed at which large-model applications spread cannot sustain that momentum, how will the model-development race evolve?
That is another question that some people have already begun to think about.
