Anthropic Co-founder: Building Claude Code, Lessons From GPT-3 & LLM System Design

Overview

Tom Brown, co-founder of Anthropic, shares his journey from early startups to pivotal contributions at OpenAI and Anthropic. The episode explores his unconventional career path, the discovery of scaling laws, the development of Claude Code, and insights into the future of AI infrastructure and innovation.

Notable Quotes

- It's kind of like a dog waiting for food to be fed to them in their bowl. At startups, you're more like wolves—you have to hunt your food, or your kids will starve. - Tom Brown, on the mindset shift from big tech to startups.

- Humanity is on track for the largest infrastructure buildout of all time, bigger than the Apollo and Manhattan projects combined. - Tom Brown, on the scale of AGI compute infrastructure.

- Anthropic's slogan is 'do the stupid thing that works.' Scaling laws were very clearly the stupid thing that worked. - Tom Brown, on the pragmatic approach to AI development.

🚀 Early Career and Startup Lessons

- Tom Brown reflects on his early days in startups, emphasizing the value of autonomy and risk-taking. He likens startup life to being a wolf, where survival depends on initiative and adaptability.

- His first startup experience at Linked taught him the importance of figuring things out independently, contrasting it with the structured environment of big tech.

- Grouper, a dating app he worked on, failed due to competition from Tinder, which solved the same problem more effectively. This taught him the importance of aligning product solutions with user needs.

📈 Scaling Laws and GPT-3 Development

- Tom Brown played a key role in scaling GPT-3 at OpenAI, transitioning from TPUs to GPUs for better software compatibility and faster iteration.

- The discovery of scaling laws—reliable intelligence gains with increased compute—was transformative. He describes the phenomenon as 12 orders of magnitude, a scale rarely seen in computer science.

- Despite criticism for brute-forcing solutions, scaling laws proved foundational for modern AI advancements.

🤖 Building Anthropic and Claude Code

- Anthropic began with seven co-founders during COVID, focused on AI safety and transformative AI. Early efforts included building training infrastructure and securing compute resources.

- Claude Code emerged as an internal tool to assist Anthropic engineers, later becoming a standout product for coding tasks. Its success was driven by empathy for Claude as a user and a focus on developer needs.

- The team prioritized internal benchmarks and qualitative evaluations over public metrics, fostering innovation without gaming external tests.

🌍 AI Infrastructure and Compute Challenges

- Tom Brown highlights the unprecedented scale of AGI compute infrastructure, growing at 3x per year. He predicts bottlenecks in power availability, especially in the U.S., and advocates for nuclear and renewable energy solutions.

- Anthropic's strategy of using GPUs, TPUs, and Trainium chips provides flexibility but requires significant performance engineering.

- He emphasizes the importance of robust software stacks to enable fast experimentation and reliable systems at scale.

💡 Advice for Aspiring AI Engineers

- Tom Brown encourages young engineers to take risks and pursue projects that align with their intrinsic motivations. He advises against chasing credentials and instead focusing on work that excites and challenges them.

- For those transitioning into AI research, he recommends self-study, including courses, Kaggle projects, and hands-on experimentation with GPUs.

- He sees opportunities in developing tools that empower AI models as productive users within human-designed systems.