u/Tobio-Star

TLDW:

1- Alignment would also need to be continual

Since there isn't really a state where the model is "finished", AI companies would need to set up frequent safety tests (maybe quarterly or monthly)

2- Research on new alignment ideas would be needed

Right now, the problem of alignment is "how to make sure that a set of weights behaves properly once deployed". In the future, we would need to answer the much difficult question "how to guarantee that, even with constant weight updates, the AI never falls prey to jailbreaks or changes into an evil persona".

My opinion: the model probably needs to have some subset of weights that aren't affected

3- The diversity of AIs will skyrocket, even if they come from the same company

4- The benefits of a superior model will compound over time

If everyone is using a model because it's the best, then that model will recursively become better

5- More pressure to release frontier models early (consequence of point 4)

6- Experience will become the new moat

Right now, it's really difficult for AI companies to differentiate themselves. In the future, it'll be really difficult to switch AIs within one's workflow because of the experience advantage that the first AI will have. Similar to how there is a high cost to firing an employee and having to train a new one for 6 months to become as competent

7- AI companies will be willing to reduce their costs to gain experience

Since experience will become maybe as valuable as the initial model's creation itself, AI companies might use carrots and sticks to gain experience: reduce costs for individuals and enterprises willing to let the models train on their data, and refuse to give frontier models to those who don't

8- Continual learning per company instead of per user

GPUs are more efficient when they run multiple requests from the same underlying model. Therefore, instead of setting up continual learning for each individual user, it might make more sense to serve an entire employee batch at once. In other words, there wouldn't be a different model for each user (which is the case in the traditional sense of CL) but one model continually learning from the entire company's data at once

My opinion: this one was a bit harder to understand for me, so take my summary with a grain of salt

---

SOURCES:
Video: https://www.youtube.com/watch?v=iewm45atodE
Original text: https://www.dwarkesh.com/p/era-of-continual-learning

Ilya is one of the most famous AI researchers ever. As recently as late 2025, he seemed very hesitant about what the next step in AGI research should be.

With that in mind, what are your expectations for this release?

Obviously I don't think he has achieved AGI so I am looking for a solid proof of concept from him. For instance, sample efficiency was an obsession of his in his 2025 podcast with Dwarkesh. So if he came up with a convincing learning algorithm that made significant strides in that direction, I would be the happiest man alive.

TLDR: The father of RL is working on an ambitious architecture, OaK, that would work with only 20 watts of energy. This feat could be accomplished through an event-driven neural net where a small part of the network is active only when a significant event occurred. OaK learns from each incoming signal (batch-size-1), eliminating the need to store large batches and reducing memory costs.

---

Richard Sutton has been talking about a grand architecture for intelligence for the past year or two, which he's labeled "OaK", short for "Options and Knowledge". It's a proposed blueprint for AGI that relies on dynamic RL where an AI learns continuously with essentially zero pretraining.

The AI would build its own concepts, those concepts would become new things to pay attention to while interacting with the world (new "Options"), and it would use those learned options to plan and improve over time.

They have 2 really ambitious goals:

• Pure in-real time learning without hacks

>Our batch-size-one learning algorithms can learn directly from experience without storing or replaying data. These algorithms, when combined with event-driven neural networks, result in systems that learn using multiple orders of magnitude less compute and energy

• Learns from noisy data with zero curation

>By learning to assign credit to parameters that generalize well, instead of assigning credit to all parameters, our methods can learn directly [and efficiently] from noisy data streams

Here is a full talk on the architecture:

Rich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025 - YouTube

Khurram Javed said that a successful prototype of OaK in the next few years would be closer to a baby learning during its first year of life than any current AI system: https://x.com/kjaved_/status/2076663868160459214

Thanks to u/Mindrust for the original thread!

---

SOURCE: https://oaklab.ai/mission

TLDR: Can local learning rules ever compete with global ones like backpropagation? ReSU shows that with the right algorithm, they can learn equally rich and complex concepts from training data. Here, the secret sauce of ReSU neurons is to extract patterns predictive of the future within their own input!

---

➤Introduction: a few fly anecdotes

Recently, flies have been at the center of major AI feats. A few months ago, some researchers managed to build a credible simulation using real fly neurons. The virtual fly remarkably exhibited many typical fly behavior within the simulated environment.

Now, a few weeks ago, another team introduced a new learning algorithm inspired by the fly's visual system. The limited complexity of the fly's brain explains why it's such a fantastic study object for neuroscience and AI, and why it is essentially the new mouse of AI research.

➤Why do we need a new learning algorithm?

While the backpropagation algorithm has been the biggest driver of progress in AI, it is also a bit of an unsatisfactory solution. To fit in the brain, it requires the presence of a global mechanism that computes an error signal and tells every single one of our neurons how to update themselves to improve the global score. But the brain doesn't work this way. It is a very local, decentralized system that doesn't leave room for a global coordinator like that.

It's not just a matter of being biologically plausible for the sake of it. It is also hypothesized that having the right learning algorithm could make AI more sample efficient. Backprop is so inefficient, that sometimes it "wastes samples" accidentally.

➤Overview of ReSU

As already said, ReSU is a new learning algorithm: a new way to teach things to models using training data. In this architecture, neurons learn by themselves. They tweak their weights on their own, without waiting for the directives of some global loss score.

But what criteria are used to make those tweaks? ReSU neurons are constantly trying to find patterns within their own input. More specifically, they try to find patterns predictive of the future. Instead of optimizing for a global loss (like backprop) or for a local loss (like predictive coding), they are looking for temporal patterns. Taking different signals as input, they try to find the combination of those signals that is the most predictive of future incoming signals. The weight updates decisions are very time-oriented.

➤ReSU in detail

What does it mean to find "a pattern predictive of the future"?

There are two cases:

• A neuron receives a single signal

In such case, the neuron tries to model how this pixel behaves over time. It looks for temporal behaviors. For instance, "this pixel goes from black to white with this specific rhythm"

• A neuron receives multiple signals (from different other neurons)

In such case, the neuron tries to capture the right combination of those signals that is the most predictive of the future (in fact, in this paper, the 2nd most predictive combinations is also kept, but let's ignore that). In both cases, the neuron's incoming local signal(s) is the only feedback used to modify its weights on the fly.

➤The math "breakthrough" behind it all

Neurons update themselves thanks to a mathematical operation called "CCA". At each time step, the neuron receives some signal. After an arbitrary number of those steps, the neuron splits them into two subgroups: the "past" and the "future" (in reality, the entire input comes from the past since it's not possible to see the future).

Finally, a comparison is performed between those two groups to find some linear relationships. That comparison is CCA. According to the team behind this paper, CCA will always find the most informative linear relationships possible (no other technique can do better)

However, if it was just that, this architecture would be very limited because CCA can only find linear relationships within the input. So after CCA, non-linearity is introduced by using a variant of ReLU, the most famous mathematical operation used by modern AI. If the relationship found by CCA is positive (meaning that the signals received by the neuron behave similarly), then the neuron outputs a number capturing the strength of that relationship. Otherwise it outputs zero.

In summary, ReSU = CCA + ReLU (roughly).

➤Adding some biological insight

In standard neural networks, each neuron is expected to find a specific pattern within the training data, and by combining billions of them, the model develops complex representations. But, biological neurons, or at least the fly ones, differ a little bit. Many sensory neurons come in pair: one is tasked with detecting a pattern, while the other is specifically designed by nature to detect the opposite pattern (or the absence of the former pattern)! You can think of them as positive vs negative neurons.

ReSU does the same thing! Instead of naively implementing standard ReLU, it implements two versions of it: ON-ReLU and OFF-ReLU. One activates itself when CCA detects a positive relationship, while the other activates itself when CCA detects a negative relationship.

This is particularly useful for binary pieces of information: a pixel can either be present (white) or absent (black), a movement can either go from right to left or left to right, etc. Modern AI makes the bet that with enough neurons, all of those nuances can still be captured by the network but ReSU implements them explicitly.

➤Biology validates ReSU!

By analyzing the "firing" patterns of ReSU neurons, researchers discovered something remarkable: they act very similarly with real fly neurons!

This was not reverse-engineered. It happened organically! By implementing ReSU, the artifical neurons in this architecture present the same activity patterns as the L1, L2 and L3 neurons found within a fly's brain. And not only was this observable with the firing patterns but also with the weights: ReSU neurons tend to give importance to the same sensory information and listen to the same other neighboring neurons, as real-life fly neurons do.

This is a very rare instance where biology directly validâtes researchers' intuition

➤Is ReSU's learning steerable? Where is supervision?

Intuitively, since neurons tweak their weights on their own and only focus on their own input, it almost seems like they just learn whatever they want to learn without any supervision whatsoever! That's what the global loss was for afterall. How can we be sure that the network is actually learning what we want it to learn?

ReSU is a huge bet on self-supervised learning (learning without supervision). The network isn't designed to learn one task in particular, but to develop a general enough representation of a domain, so that such a representation can work for any task of said domain. The hope is that the model extracts as much informative feature from training data as possible.

If ReSU, or a ReSU-like idea turns out to be the right way to build intelligent models in the future, then ReSU would serve as the self-supervised learning phase, while a subsequent fine-tuning phase would provide explicit supervision (though for now, the compatibility between these 2 steps hasn't been figured out).

➤Emergence of useful complex representations

Local learning algorithms have always hit the same wall: learning useful complex representations. This is another consequence of the lack of supervision.

Since local learning algs do not rely on a global loss, the 1st layer of the network is prone to learning useless stuff that the subsequent layers build on, dooming the entire chain of representation. Thus the entire hierarchical representation learned by the network can be completely useless. Backprop avoids this because the 1st layer is always kept in check with the global loss score

The team behind ReSU is making another bet here: that by extracting features predictive of the future, the network will inherently learn useful information. The learning rule of the neurons themselves is the supervision here, at least until a potential fine-tuning phase is added.

OPINION

This paper is interesting for many reasons. First, they leveraged biology in a very unusual way. The biological details they went to is a level that AI researchers usually don't touch, and it is very impressive.

Second, they confronted a problem that at least to my knowledge, proponents of local learning algorithms usually don't explicitly acknowledge: making sure that the model learns useful hierarchical representations. I never knew why exactly something like predictive coding still isn't widely adopted by the AI community (outside of "backprop already works"). Now I know.

In general, the sheer amount of work that went into this paper deserves a lot of respect

SOURCES: 
Paper: https://arxiv.org/abs/2512.23146
Thumbnail: https://neurosciencenews.com/fly-brain-model-neuroscience-3227/

TLDR: While AI is being taken increasingly seriously, very little progress has been made on sample efficiency. The amount of data these models rely on is so unfathomable, that once one fully grasps its scale, it becomes obvious that even the very idea of an AGI timeline might as well be fantasy without serious efforts in fundamental research.

---

➤The observation

Currently, AI has a massive sample efficiency problem. Even the tiniest variation of tasks can only be solved by a data black hole: trillions of tokens on which LLMs were trained to solve all kinds of questions. Instead of relying on intuition and common sense like a human would, we've created a Frankenstein-like, barely sewn-together monster of data to deal with coding, math, medicine, or even some random software.

➤Two sides of the same coin

Really, the problem can be seen in 2 ways:

1- We need data for literally every single possible task. Even if the model masters 30 programming languages, learning a slightly new variant requires going back to training. It doesn't matter if it shares the same fundamental concepts. The same applies to any random software.

2- We need a gargantuan amount of said data. So not only do we need to train the model for every piece of software under the sun, we also need ridiculous amounts of data for EACH of them.

Hundreds of human experts are tasked with writing tens of examples for every single part of their workflow. It would be like an educated human needing hundreds of professors just to learn to correctly format a word document.

➤RL to generate even more data

RL is not only used to teach models to solve math or coding problems. It is also used to generate even more data. Each time a model successfully solves a task through trial-and-error, the reasoning traces themselves become training material.

This overabundance of data seems like the antithesis of what AGI should be. General intelligence has always been about generalizing out of distribution. Being able to learn new skills with minimal examples. Not a weird patchwork of unrelated skills. The generalization abilities of these models are at best fragile.

➤Can sample efficiency be scaled?

To some extent yes. It has been demonstrated that bigger models, i.e. models with more parameters, learn new skills faster. They need less data. Almost as if they had more computing power to search for the algorithm that underlies the training data.

However, that effect is limited. The scaling laws show that even if we took GPT5 and increased its number of parameters to INFINITY, the amount of data it would need to learn, say, a new programming language would at best decrease 10 fold. In other words, if current GPT5 needs 100k tokens to learn C++, increasing its parameters to infinity would take that down to 10k tokens... which is still an absurd amount.

By contrast, humans are millions of times more sample efficient than these models, suggesting that our brains follow a different scaling curve altogether. The architecture of the human brain is inherently smarter than these models, and by a lot.

➤Could evolution explain the discrepancy?

Using evolution to dismiss observations on sample efficiency is very common in this field.

2 arguments tend to resurface:

1- The human genome

The genome is only about 3GB of data. That is simply not enough to store meaningful amounts of world knowledge. At best, it is hypothesized that the genome contains the brain's hyperparameters and loss functions, to tell us what we should pay attention to while interacting with the real world. Barely any knowledge is encoded there.

2- Multimodal data

Some people suggest that even if humans don't rely on text, we probably rely on sensory data that is just as informative as text if not more so. Dwarkesh counters this argument by citing blind and deaf people who are still generally intelligent while barely having any sensory tokens to rely on.

Personally, I would disagree with Dwarkesh slightly here. Most humans, including blind and deaf people, can "feel" the environment through touch and motion, allowing us to develop complex notions such as shape and space, which are at the heart of our reality (almost every single field, including math or even coding, involves concepts from these 2 notions in some way).

However, this only shifts the problem: multimodal data is clearly a massive weak point of current AI, and is a very hard research problem. Some of the dumbest animals on planet earth have a much better understanding of space and shape than our top models. And top of being seemingly as hard to solve as finding a general cure for cancer, the industry doesn't always care that much about multimodality. Case in point: Anthropic has basically chosen to ignore anything that isn't text-based.

➤Does sample efficiency really matter?

While humans learn much faster than AI, we are profoundly limited in the amount of data we can handle both at once and throughout our lifetime. AI can learn in parallel, and is fast enough to (at least theoretically) read the entire internet at once. Humans do not have that ability.

AIs can also merge their brains together to share their knowledge with other models, something we fundamentally cannot do. What if, by continuing to bet on AIs' strengths, they end up making up for their deficiencies in the long run? Or maybe AI could speed up AI research itself!

Dwarkesh seems somewhat skeptical of these arguments because it's essentially betting that systems with brittle generalization could somehow figure out a problem so difficult, and so out-of-distribution, that even humans still cannot solve it!

---

OPINION

Dwarkesh has really opened my eyes on how reliant AI is on data. After hearing his arguments, it is mind-boggling to me how such a significant portion of the field can believe AGI to be 2 years away while even a fully trained model still needs absurd amounts of data to learn any simple piece of software.

I think all of this highlights why common sense is important in research. We shouldn't just rely blindly on metrics and benchmarks. If my model needs ridiculous amounts of data for every little variation of a task, or if it fails basic common-sense questions, why should I care about its results on math benchmarks when math supposedly involves far more complex concepts?

Metrics are a useful crutch to assess the intelligence of these models, but imo the overall evaluation should rely on a mix of local, common sense-based experiments along with these huge evals.

SOURCE: https://www.youtube.com/watch?v=4pG3SJQPAwk

TLDR: Scaling has always been mostly about increasing the total number of neurons in a neural network. But the biological neuron is infinitely more complex than artificial ones. What if we also scaled internal neuron complexity? This paper provides quantitative evidence for doing so
---

➤Towards more biomimetic neurons

Current AI has relied on a massive number of trivially simple neurons, and the results have been spectacular thus far. But as we hit some performance walls, a group of researchers tried answering the following question: could scaling the internal neuron complexity be a new scaling axis for AGI?

The researchers evaluated different neural networks on 3 scaling axes: total number of neurons, total number of connections, and, newly, internal neuron complexity. The relationship between compute and these 3 variables respectively follows P = N(ke + kc). In other words:

• investing only in neuron count is always leaving some meat on the bone. The optimum always involves a fine balance between network size (neuron count), neuron complexity and connectivity.
• as compute budget grows (defined as the total number of parameters), the optimal architecture shifts toward both larger networks, more complex neurons, and higher connectivity

Note: after a certain point, scaling neuron complexity also hits diminishing returns because each neuron is already extracting as much information as possible

➤The overlooked role of recurrence

Recurrence simply means that a network's current state depends on its past states, which implies keeping track of time and maintaining some temporal memory. This is hypothesized to be important because the world is both deeply temporal (eg. video and audio) and sequential (eg. text).

The brain is massively recurrent. Its sensitivity to time is reflected in our tendency to focus on changes while gradually ignoring constants. That's why we can tune out background noise and still notice new sounds.

In neural networks, recurrence can be achieved by increasing the number of connection loops so that neurons communicate back and forth with each other. Neuron A (or group of neuron A) is connected with Neuron B which is connected back to Neuron A. There are tons of this kind of loop in the brain

On top of making us more time-aware, scaling the number of connections also reduces redundancy: the more neurons communicate with each other, the more they'll be incentivized to learn different things.

➤Inside the ELM ("Expressive Leaky Memory") architecture

This architecture is focused on implementing both recurrent and expressive neurons.

-Recurrence

The authors implemented recurrence in two ways:

1- they manually connected neurons in order to force them to do a lot of loops between each other

2- their internal state is recurrent: the current state of a neuron depends on its past

-Expressiveness

A classical neuron takes input from surrounding neurons, sums it, and passes the result through a nonlinear activation function. ELM neurons are far more complex. Each of them are like whole dynamical ecosystems:

1- At time t, incoming signals are first split into groups and processed through branch-like structures loosely inspired by dendrites. This delays the mixing of information and allows the model to capture more complexity within the input

2- The processed input is compared against the neuron's internal memory through a small MLP to compute a memory update. This memory is itself composed of multiple smaller memory units operating on different timescales (milliseconds, seconds, minutes, hours...)

Note: Scaling neuron complexity usually means increasing the size of this internal MLP and the number of those smaller memory units.

3- The resulting memory update is merged with the previous memory to produce a proposed output. But this is not yet the final output. This proposal still has to be compared to an average of the neuron's past outputs before deciding on the final output at time t+1

This step's goal is to explicitly make the neuron sensitive to changes rather than raw output. A bit like how a human's brain gets used to some background noise and only pays attention when it hears a new sound. The ELM neuron pays attention to changes instead of constants by tracking its own activity pattern.

➤Results

The biomimetic ELM architecture performs quite well on spiking audio benchmarks as well as a modified Wikipedia corpus. It's nowhere near replacing Transformers as that was never the point, but it suggests that implementing both expressive and recurrent neurons could truly unlock AI

---
PAPER: https://arxiv.org/abs/2605.12049

They seemed like the next logical step for LLMs, with extraordinary speed benefits. Google Diffusion had decent marketing too.

I know that diffusion models can be less practical because some applications really require autoregressiveness (text-to-speech, software that does something for every new word received instead of waiting for the complete sequence), but I am still really surprised by the lack of news and development on this.