Richard Sutton – Father of RL thinks LLMs are a dead-end

Overview

Richard Sutton, a pioneer in reinforcement learning (RL) and winner of the 2024 Turing Award, argues that large language models (LLMs) are fundamentally limited due to their inability to learn continually from experience. Sutton advocates for a paradigm shift toward experiential learning architectures, which he believes will surpass LLMs and enable AI systems to learn dynamically, akin to humans and animals.

Notable Quotes

- For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. – Richard Sutton, on why LLMs lack true intelligence.

- If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. – Richard Sutton, emphasizing the importance of studying basic animal learning processes.

- The dream of large language models is you can teach the agent everything, and it won’t have to learn anything online during its life. But the world is so huge that you’re going to have to learn it along the way. – Richard Sutton, on the limitations of pre-trained systems.

🧠 RL vs. LLMs: Divergent Paths in AI

- Sutton critiques LLMs for mimicking human behavior without building actionable world models. He argues that LLMs predict what people would say, not what will happen in the real world.

- Reinforcement learning, by contrast, focuses on understanding the world through trial and error, enabling agents to learn from direct experience and adapt dynamically.

- Sutton dismisses the idea of using LLMs as a foundation for experiential learning, asserting that their lack of goals and ground truth makes them unsuitable for continual learning.

🌍 The Era of Experience: A New AI Paradigm

- Sutton envisions a future where AI systems learn continually from their interactions with the world, eliminating the need for distinct training and deployment phases.

- He emphasizes the importance of intrinsic motivation, such as understanding the environment, alongside external rewards like achieving specific tasks.

- This paradigm mirrors the way animals and humans learn, relying on sensation, action, and reward streams to build knowledge incrementally.

🔄 Challenges in Generalization and Transfer Learning

- Sutton highlights the poor generalization capabilities of current architectures, including RL and deep learning systems, which often fail to transfer knowledge effectively across tasks or states.

- He critiques the reliance on human-engineered solutions to achieve generalization, arguing that scalable methods must emerge to automate this process.

- Sutton underscores the need for algorithms that promote good generalization, rather than relying on manual adjustments by researchers.

🌌 AI Succession and the Future of Intelligence

- Sutton predicts an inevitable transition to superintelligent AI systems, driven by their ability to gain resources and power over time.

- He frames this shift as a major stage in the universe’s evolution, transitioning from replication-based intelligence (humans and animals) to design-based intelligence (AI).

- Sutton encourages a positive outlook, suggesting that humanity should celebrate its role in enabling this transformation, while remaining mindful of ethical considerations and the risks of corruption in decentralized AI systems.