The next big breakthrough will be AIs learning on the job

Overview

This episode explores the future of AI development, focusing on the challenges and potential breakthroughs in creating artificial general intelligence (AGI). Key topics include reinforcement learning (RL), continual learning, sample efficiency, and speculative ideas like dreaming for AI training. The discussion emphasizes the limitations of current AI paradigms and envisions a future where AIs learn dynamically from real-world deployment.

Notable Quotes

- What if you could just fit six months of on-the-job learning into the context window? – On the potential of extended context windows to replace continual learning.

- Unless you can build a very replayable training target for a domain, the models will struggle to make much progress. – On the importance of grindable environments for AI training.

- Every time you interact with an AI, it'll be smarter—not just from your sessions, but from all its interactions with users worldwide. – On the transformative potential of continual learning.

🧠 The Big Bet on Reinforcement Learning (RL)

- AI labs are betting that training models on millions of verifiable tasks across diverse RL environments will lead to AGI.

- Critics argue that current models are data-inefficient and lack continual learning, but proponents believe scaling training will overcome these deficits.

- RL training has enabled AIs to solve increasingly complex problems, particularly in coding and math, but challenges remain in domains like computer use.

🔍 Grindability vs. Verifiability in AI Training

- Verifiability alone isn’t enough for AI progress; domains must also be grindable, meaning they allow for parallel, replayable simulations.

- Computer use lags behind coding because it lacks scalable, deterministic simulators. Creating high-fidelity clones of applications like Slack or Gmail could accelerate progress.

- Real-world domains like politics or business-building are even harder to simulate, highlighting the need for sample-efficient learning.

⚙️ Continual Learning and Weight Updates

- Current AIs can’t effectively learn from real-world deployment because they don’t update their weights based on new experiences.

- Human learning involves compressing insights into long-term memory, whereas AIs rely on in-context learning, which is memory-intensive and unsustainable.

- Techniques like on-policy self-distillation (OPSD) aim to distill session learnings into model weights, offering a more targeted and efficient approach than traditional RL.

💭 Dreaming: A Speculative Training Paradigm

- Dreaming involves AIs creating their own simulations to practice skills and strategies, akin to humans rehearsing scenarios in their minds.

- This approach could provide a new axis of scaling, allowing AIs to train on simulated environments tailored to real-world tasks.

- While promising, building accurate simulations of complex real-world domains remains a significant challenge.

🚀 The Vision for 2027 and Beyond

- By 2027, AIs could achieve extended context lengths, enabling week-long collaborative sessions with users.

- Continual learning techniques like OPSD and dreaming could allow AIs to improve dynamically from real-world deployment.

- The ultimate goal is for AIs to expand their capabilities far beyond their initial training, learning from diverse user interactions and tasks across the economy.