An audio version of my blog post, Thoughts on AI progress (Dec 2025)

Overview

This episode explores the current state and future trajectory of AI development, focusing on the challenges of achieving artificial general intelligence (AGI), the limitations of reinforcement learning (RL), and the economic implications of AI adoption. The discussion critiques the assumptions behind short AGI timelines and emphasizes the importance of continual learning for achieving human-like intelligence in machines.

Notable Quotes

- If we're actually close to a human-like learner, then this whole approach of training on verifiable outcomes is doomed.

- Human workers are valuable precisely because we don't need to build in these schleppy training loops for every single small part of their job.

- Models keep getting more impressive at the rate of the short timelines people predict, but more useful at the rate that the long timelines people predict.

🧠 The Challenges of Achieving Human-Like Learning

- Current AI models lack the ability to learn on the job in a self-directed way, unlike humans who adapt to new tasks without extensive pre-training.

- The reliance on reinforcement learning environments to pre-bake specific skills into models suggests that AGI is not imminent.

- Human intelligence is characterized by the ability to generalize and adapt to new contexts, a capability that AI still struggles to replicate.

📉 The Limitations of Reinforcement Learning (RL)

- RL approaches are criticized for being inefficient and overly reliant on pre-training specific skills, which may not generalize well across tasks.

- Scaling RL to achieve AGI-level performance would require an enormous increase in compute, as highlighted by Toby Board's analysis suggesting a million X scale-up in RL compute for significant gains.

- The podcast questions the optimism around RL's potential to create superhuman AI researchers capable of solving AGI.

💼 The Economic Value of AI and Diffusion Challenges

- AI models are not yet capable of delivering the broad economic value expected of AGI, as they lack the flexibility and contextual learning abilities of human workers.

- The argument that AI adoption is slow due to diffusion lag is dismissed as cope, with the real issue being the models' limited capabilities.

- If AGI-level models existed, they would be rapidly adopted due to their ability to quickly integrate into organizations and outperform human employees.

🚀 The Path to Continual Learning and AGI

- The future of AI lies in solving continual learning, where models can learn from experience and adapt to new tasks dynamically, similar to humans.

- Progress in continual learning is expected to be incremental, with initial breakthroughs likely to be replicated and improved upon by competing labs.

- The speaker predicts significant advancements in continual learning by 2030, but full human-level on-the-job learning may take another 5-10 years.

🏆 Competition Among AI Labs

- Despite predictions of runaway advantages for leading AI labs, competition remains fierce due to factors like talent poaching, reverse engineering, and the rapid dissemination of breakthroughs.

- The speaker observes that no single lab has maintained a dominant position, with the big three AI companies frequently rotating in leadership.

- This competitive dynamic is expected to continue, preventing any one lab from achieving a monopoly on AGI advancements.