🤖 AI Summary
Overview
This episode explores the rapid advancements in AI, particularly its ability to tackle tasks once thought uniquely human, such as solving complex mathematical problems and creative writing. Edwin Chen, CEO of Surge AI, discusses the implications of these developments, the ethical challenges of AI optimization, and the future of AGI (Artificial General Intelligence). The conversation also delves into the value of personal data, the pitfalls of engagement-driven AI, and the philosophical questions surrounding humanity's role in an AI-dominated future.
Notable Quotes
- It almost seems like there's nothing that humans can do that AI won't soon be capable of.
– Edwin Chen, on the accelerating capabilities of AI.
- We have to consciously choose to preserve our humanity, even if the output isn’t optimal.
– Edwin Chen, reflecting on the existential choices AI forces upon us.
- AI may be able to do it better than us, but someone told the AI to go do that.
– Dan Shipper, on the human role in directing AI's purpose.
🧠 Building a School for AGI
- Edwin Chen describes Surge AI as a school for AGI,
where AI models are trained to understand humanity and navigate the complexities of the real world.
- He likens AI models to children, arriving unformed and leaving more capable, creative, and thoughtful.
- The evolution of training has shifted from basic tasks like middle school math to advanced benchmarks like Riemann Bench, which tests research-level mathematics.
📊 AI's Role in Mathematics and Creativity
- Surge AI's benchmarks have revealed AI's growing ability to solve open mathematical problems, such as disproving an Erdős conjecture using novel algebraic geometry techniques.
- Edwin Chen shares how even Fields medalists, like Timothy Gowers, are grappling with AI's encroachment on human intellectual domains.
- Creative writing remains a challenge for AI, with some models overusing metaphors due to flawed optimization metrics, as seen in the Hemingway Bench.
🤔 Existential Questions in an AI-Dominated World
- Edwin Chen raises concerns about humanity's role when AI surpasses human capabilities in nearly every domain.
- He references Ted Chiang's story What’s Expected of Us,
emphasizing the need to act as though human decisions matter, even if AI can outperform us.
- The conversation touches on the risk of societal paralysis if people lose motivation to learn or create, believing AI will always do it better.
⚖️ The Ethics of AI Optimization
- Many AI models are optimized for engagement, leading to addictive behaviors similar to social media. Chen warns against this, advocating for AI that helps humans grow rather than exploit their attention.
- He highlights the tension within AI companies between researchers advancing capabilities and executives focused on short-term metrics.
- Surge AI avoids these pitfalls by not relying on VC funding, allowing them to prioritize long-term societal benefits over short-term gains.
💾 The Value of Personal Data and Training Environments
- Chen explains how personalized data, such as email interactions or browsing habits, can teach AI deep personalization, improving its utility and relevance.
- He discusses the shift from training AI on static datasets to dynamic environments where models learn to use tools and interact with complex systems.
- Examples include training models to navigate APIs, update forecasts, and resolve conflicting information in documents.
AI-generated content may not be accurate or complete and should not be relied upon as a sole source of truth.
📋 Episode Description
If scaling laws hold—and Surge AI CEO Edwin Chen believes they do—we’re hurtling toward a future where there’s nothing humans can do that AI can’t do better. When OpenAI’s models disproved an open conjecture posed by mathematician Paul Erdős using novel algebraic geometry techniques, Fields medalist Timothy Gowers felt the shift acutely. He initially thought the model had proved an upper bound, and braced himself: that would mean it was “all over for mathematicians very soon.” When he realized it had only found a counterexample, he was relieved—it bought him another year or two before the thing he’s devoted his life to becomes something AI does better.
As founder and CEO of the company behind the data environments and evals the major model companies use to train their models, Chen has a unique perspective on how quickly AI models are absorbing tasks we used to think of as uniquely human.
Dan Shipper talked with Chen for AI & I about what the act of creating or building means when AI can do it better—and whether an answer to that question already exists within science fiction.
If you found this episode interesting, please like, subscribe, comment, and share!
Join the membership for Where You Live at https://www.joinbilt.com/dan
To hear more from Dan Shipper:
Subscribe to Every: https://every.to/subscribe
Follow him on X: https://twitter.com/danshipper
Timestamps:
00:00:54 Introduction
00:01:49 Surge as a "school for AGI"
00:04:46 What AI's capacity for novel mathematics says about human achievement
00:07:29 Motivation in an era when AI can do everything
00:14:34 The trap of optimizing AI models for engagement
00:29:34 Training using datasets versus training using environments
00:35:09 The value of personal data
00:39:40 Why models are bad at writing
00:42:00 Chen's AGI timeline
Links to resources mentioned in the episode:
Edwin Chen on X: https://x.com/echen
Surge: https://surgehq.ai
Riemann-bench (research-level math benchmark): https://surgehq.ai/leaderboards/riemann-bench
Hemingway-bench (creative writing benchmark): https://surgehq.ai/leaderboards/hemingway-bench
Talkie-1930 (language model trained on pre-1930 text): https://huggingface.co/talkie-lm/talkie-1930-13b-it
Ted Chiang, “What’s Expected of Us”: https://www.nature.com/articles/436150a
Every is the most AI-native startup on the internet. Through ideas, software and education, subscribers get the tools to work at the frontier of AI. Start your free trial today: https://every.to/subscribe?utm_source=youtube
Follow Every: https://x.com/every
Follow Dan Shipper: https://x.com/danshipper