AI Letters #03 - The Bitter Lesson, 10 Years Later
· 3 min read
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." - Richard Sutton, 2019
Every five years or so, the AI field collectively rediscovers the same uncomfortable truth. Hand-crafted knowledge loses. Scale wins. The researchers who built the most elegant domain-specific systems watch a much simpler model trained on more data walk past them.
Sutton called it in 2019. GPT-3, AlphaFold 2, Sora, and now reasoning models have confirmed it repeatedly since. The pattern is not a coincidence - it is a structural property of the problem.
This letter was first published on AI Letters →
Continue reading on Substack →
Interactive Data Stories
Interactive Chart
Domain Timeline →
When scale beat knowledge engineering in Chess, NLP, Vision, Proteins, and Robotics - the full history, domain by domain.
Interactive Chart
The Two Paradigms →
Move the compute slider - watch Knowledge Engineering hit its ceiling while Scale follows a power law indefinitely.
Interactive Chart
The Evidence →
GPT-3, AlphaFold 2, Stable Diffusion, Sora - the numbers, dates, and scale that proved Sutton right.