ElevenLabs’ Mati Staniszewski: Why Voice Will Be the Fundamental Interface for Tech

Overview

Mati Staniszewski, co-founder of ElevenLabs, discusses how the company has revolutionized text-to-speech technology by focusing on contextual understanding, emotional delivery, and voice replication. He shares insights into the technical challenges of building voice AI, the viral moments that propelled ElevenLabs into the spotlight, and the transformative potential of voice as a primary interface for technology. The conversation also explores the future of real-time translation, voice agents, and the cultural implications of breaking language barriers.

Notable Quotes

- Voice will fundamentally be the interface for interacting with technology—it carries emotions, intonation, and imperfections that text simply cannot. – Mati Staniszewski

- We hope to cross the Turing test for voice interactions this year, making it indistinguishable from speaking to another human. – Mati Staniszewski

- The biggest barrier to global communication is not understanding the other person. Real-time voice translation will change the world. – Mati Staniszewski

🎙️ The Origins of ElevenLabs

- The idea for ElevenLabs stemmed from a frustrating experience watching a Polish-dubbed movie, where all characters were voiced by a single monotonous narrator.

- Co-founders Mati and Piotr, friends since high school, bonded over mathematics and later collaborated on hack projects, including early experiments in audio AI.

- Inspiration came from advancements like the Attention Is All You Need paper and open-source models like Tortoise TTS, which hinted at the potential for high-quality voice replication.

🛠️ Building a Defensible Position in Voice AI

- ElevenLabs focused narrowly on audio, avoiding competition with multimodal foundation models.

- Early innovations included applying transformer and diffusion models to text-to-speech, enabling nuanced emotional delivery and contextual understanding.

- Challenges included sourcing high-quality audio data with accurate transcriptions and capturing non-verbal elements like tone and emotion.

- The company developed unique voice encoding techniques, allowing models to replicate voices without hardcoding features like gender or age.

🌍 Breaking Language Barriers and Real-Time Translation

- ElevenLabs aims to eliminate language barriers by enabling real-time voice translation while preserving the speaker's unique tone and emotion.

- Viral use cases include dubbing Lex Fridman’s interview with Prime Minister Modi into Hindi and English, and creating multilingual narration for European languages.

- Staniszewski envisions a future where people can travel and communicate seamlessly in any language, likening it to the Babel fish from The Hitchhiker's Guide to the Galaxy.

🤖 The Rise of Voice Agents

- Voice agents are becoming a popular interface for tasks like customer support, healthcare, and education.

- Examples include automating nurse-patient calls, creating chess tutorials narrated by iconic players, and enabling interactive journalism with Time magazine.

- ElevenLabs powers voice interactions for gaming (e.g., Darth Vader in Fortnite) and enterprise applications, emphasizing quality, low latency, and scalability.

- The company is working on duplex models to improve contextual responsiveness and reduce latency further.

🇪🇺 Building a Global Company from Europe

- Advantages of being based in Europe include access to passionate, high-caliber talent and a growing enthusiasm for AI innovation.

- Challenges include fewer experienced mentors compared to the U.S. and regulatory hurdles like the EU AI Act.

- ElevenLabs’ global mindset from inception has helped it cater to diverse languages and cultures, aligning with its mission to make audio universally accessible.