Richard Sutton

The Man Who Kept Insisting on the Obvious

There’s a particular kind of intellectual contribution that looks trivial in retrospect and heretical at the time. Richard Sutton has made a career of this. He didn’t invent reinforcement learning from scratch — the conceptual roots stretch back through optimal control theory, dynamic programming, and animal psychology — but he gave the field its modern computational form, its canonical textbook, and eventually its most uncomfortable philosophical provocation. That provocation, a short essay called “The Bitter Lesson,” published in 2019, is only a page and a half long. It has arguably reshaped more strategic thinking in AI than papers ten thousand times its length.

Reinforcement Learning as a Foundational Frame

Sutton’s core technical contribution is the formalization of reinforcement learning (RL) as a distinct computational paradigm. The key insight, developed through the 1980s and 1990s alongside Andrew Barto and others, was that learning from interaction — from reward signals rather than labeled examples — constituted a fundamentally different problem from supervised learning. The agent doesn’t get told what the right action was. It gets told how well things went. This seems like a small distinction until you realize it’s the distinction between a student correcting an exam and a creature navigating a life.

Sutton’s 1988 paper on temporal-difference (TD) learning is the technical fulcrum. TD methods learn to predict future rewards by bootstrapping — using current estimates of future value to update past estimates. This is deeply unintuitive the first time you encounter it: the algorithm improves its predictions by comparing them not to actual outcomes but to its own subsequent predictions. It’s like calibrating a clock by checking it against itself tomorrow. And yet it works. TD(λ) unified Monte Carlo methods (which wait for full outcomes) with pure bootstrapping (which doesn’t wait at all) along a continuous parameter λ, giving practitioners a dial between variance and bias.

The Sutton & Barto textbook, Reinforcement Learning: An Introduction (first edition 1998, second edition 2018), became the canonical onramp. What makes it exceptional isn’t just its clarity but its honesty about what remains unsolved. It presents RL not as a finished theory but as a productive formalism — a way of asking questions about agents, environments, value, and policy that makes certain kinds of progress possible.

The Bitter Lesson

Then there’s the essay. “The Bitter Lesson” (March 2019) argues, with the force of a thesis nailed to a door, that 70 years of AI research point to one conclusion: general methods that leverage computation scale better than methods that leverage human knowledge. Chess programs improved not because we got better at encoding grandmaster intuition but because search got faster. Speech recognition improved not because linguists built better phoneme models but because statistical methods ate more data with more compute. Computer vision improved not because we engineered better edge detectors but because deep learning scaled.

The lesson is “bitter” because it means that much of what AI researchers do — the careful feature engineering, the domain-specific heuristics, the lovingly crafted representations — is eventually made obsolete by brute-force methods that don’t care about human understanding. Researchers keep making the same mistake, Sutton argues, because encoding human knowledge feels like progress. It is progress, in the short term. But it creates a ceiling. Computation-leveraging methods have no ceiling — or at least, no ceiling we’ve hit yet.

This is a stronger claim than it appears. It’s not just an observation about what has happened; it’s a prescription about what researchers should do. Stop trying to build in what you know. Build systems that can find things out. The implication is that the intellectual virtues AI researchers most prize — deep domain expertise, elegant representations, theoretical insight into problem structure — are, in the long run, less important than the engineering virtues of scalability and generality.

Where It Connects and Where It Frays

The Bitter Lesson resonates far beyond AI. It echoes similar tensions in economics (planning vs. markets), epistemology (rationalism vs. empiricism), and even biology (intelligent design vs. evolution). In each case, the question is whether top-down knowledge or bottom-up search produces better outcomes. Sutton is firmly on the side of search, and the LLM revolution has made his position look almost prophetic.

But the lesson has limits that are worth probing. Sutton acknowledges that computation-leveraging methods still require some human choices — the architecture, the loss function, the training protocol. The question is how much of the total contribution these choices represent. Scaling laws research (Kaplan et al., Hoffmann et al.) has made this partly empirical: it turns out that architecture matters less than scale, but it doesn’t matter zero. The Bitter Lesson might be 85% right, which is importantly different from 100% right.

There’s also a temporal subtlety. In any given decade, human-knowledge-leveraging approaches often win. The bitter lesson only holds if you zoom out far enough. This creates a real tension for practitioners: should you build what works now, or invest in what will work in twenty years? Sutton’s answer is clear, but it’s the answer of someone with a theorist’s time horizon, not an engineer’s deadline.

And there’s a deeper philosophical question the essay doesn’t fully address: what is computation leveraging for? If the goal is to build systems that perform well on benchmarks, the bitter lesson is decisive. But if the goal is to understand intelligence — to know why the system works, to build models that illuminate the structure of cognition — then the lesson offers less guidance. Sutton himself has been deeply interested in understanding, not just performance. His more recent work on the “Alberta Plan” envisions AI agents that build world models, maintain predictions at multiple timescales, and construct something like understanding from experience. This is not the work of someone who thinks computation alone is sufficient. It’s the work of someone who thinks computation is necessary but wants to aim it at the right problems.

What Remains Live

The most interesting unresolved thread in Sutton’s legacy is the relationship between RL and the current deep learning paradigm. Modern LLMs are primarily trained with supervised and self-supervised learning; RL enters mainly in the fine-tuning stage (RLHF and its descendants). But Sutton’s vision was always larger: agents that learn continually from interaction with an environment, that build and revise models of the world, that plan using those models. This vision hasn’t been realized at scale. The systems that dominate today are, in Sutton’s own framework, more like pattern completers than agents. Whether the next era of AI looks more like “RL agents in the world” or “ever-larger predictive models prompted by humans” remains genuinely open.

There’s also the question of whether the Bitter Lesson will stay bitter. If scaling hits fundamental physical or economic limits — energy costs, data exhaustion, diminishing returns — then the human-knowledge-leveraging approaches might have their revenge. The lesson’s truth depends on a contingent fact about the universe: that computation remains cheap enough, relative to the problems we care about, to make brute search win. That’s been true so far. Whether it stays true is not a question AI theory can answer.

Why It Matters

Sutton matters because he combined two rare things: deep technical work that actually moved a field forward, and the willingness to articulate an uncomfortable meta-lesson about the work itself. The Bitter Lesson is, in some sense, an act of intellectual self-abnegation — a researcher telling his community that much of what they do well doesn’t matter as much as they think. That takes a particular kind of clarity, and a particular kind of courage. The question he forces is the one every technical field eventually faces: are we building monuments to our own cleverness, or are we actually solving the problem? The answer, Sutton suggests, is usually both — but only one of those activities scales.