I read every major CS paper of the last 100 years...

Overview

This episode explores the evolution of computer science and artificial intelligence through ten groundbreaking research papers that shaped modern computing. From Alan Turing's foundational work on algorithms to OpenAI's GPT-3, the discussion highlights how these innovations built upon each other to create the AI-driven world we live in today.

Notable Quotes

- Alan Turing defined the machine. Claude Shannon gave it currency. Rosenblatt gave it a neuron. Jeffrey Hinton taught it how to learn. Google gave it data and an architecture. And OpenAI just turned the dial to the maximum.

- What is ChatGPT even doing? Well, it's just predicting the next word or token, just like Claude Shannon was doing in 1948.

- The crazy discovery though is that the middle hidden layers started inventing their own features—edges, shapes, and concepts that nobody programmed in.

🧠 The Foundations of Computing

- Alan Turing (1936) introduced the concept of the Turing Machine in his paper On Computable Numbers, proving that some mathematical problems cannot be solved by algorithms. This work laid the groundwork for all modern computing devices.

- Claude Shannon (1948) defined information as a measurable entity in A Mathematical Theory of Communication, introducing the bit as the fundamental unit of information and linking it to entropy. His work became the basis for data compression, prediction, and even modern AI loss functions.

🧪 The Birth and Challenges of Neural Networks

- Frank Rosenblatt (1958) created the perceptron, the first machine capable of learning from data by mimicking the brain's neurons. This sparked early AI hype.

- Marvin Minsky and Seymour Papert (1969) debunked the perceptron's limitations in their book Perceptrons, showing it couldn't solve basic problems like XOR logic. This led to the first AI winter, but they also hinted that stacking layers of perceptrons could overcome these issues.

- Jeffrey Hinton (1986) solved the training problem for multi-layer perceptrons with backpropagation, enabling neural networks to adjust weights layer by layer using calculus. This breakthrough revived interest in AI.

⏱️ Distributed Systems and Scalability

- Leslie Lamport (1978) tackled the problem of synchronizing distributed systems in Time, Clocks, and the Ordering of Events. He introduced logical clocks, allowing systems to maintain order without relying on real-time clocks. This innovation underpins modern databases, blockchains, and large-scale AI training.

🌐 The Internet and Data Explosion

- Sergey Brin and Larry Page (1998) revolutionized web search with the PageRank algorithm, described in The Anatomy of a Large-Scale Hypertextual Web Search Engine. By treating links as weighted votes, they created Google and amassed the largest structured text dataset, which later fueled AI advancements.

🤖 The Deep Learning Revolution

- Alex Krizhevsky, Ilya Sutskever, and Jeffrey Hinton (2012) demonstrated the power of deep learning with AlexNet, a convolutional neural network trained on the massive ImageNet dataset. It drastically outperformed competitors in image classification, proving that deep learning works with sufficient data and compute.

- Ashish Vaswani and Google (2017) introduced the transformer architecture in Attention Is All You Need, enabling models to process entire sequences of data simultaneously. This innovation became the backbone of modern large language models like GPT.

- OpenAI (2020) scaled transformers to unprecedented levels with GPT-3, as detailed in Language Models are Few-Shot Learners. By training on 175 billion parameters, GPT-3 demonstrated emergent capabilities like translation and coding without explicit programming.