


"Great observation by @fchollet . Frontier labs are almost doing no pretraining-- probably because pretraining is bottlenecked by data- the world is running out of data. So they focus on reinforcement learning, and there are many domains that may be insufficiently data abundant and too..."
> ...messy/unverifiable for RL. > > > — Luis Garicano
Source: https://x.com/lugaricano/status/2107853818784272623
> What if the jagged frontier is mainly math + code (which you can push arbitrarily far with RLVR), and everything else starts to plateau because it is still bottlenecked by human generated data? > > Model performance in non-verifiable areas has kept improving steadily, albeit much > > — François Chollet