ElevenLabs Enables AI to Respond in Under Half a Second
Eleven Realtime was launched yesterday: two speech models that listen, think, and respond all at once. What does that mean?
Figure 01.0 — ElevenLabs Enables AI to Respond in Under Half a Second
First, the Concepts
A voice assistant typically consists of separate components. One component converts your voice into text. A language model devises the response. A third component then turns that back into speech. Additionally, something needs to determine when you've finished speaking. Each step takes time, and together it often feels like talking to someone who is always a beat behind.
Speech-to-speech means: sound in, sound out. Without you having to connect all those components yourself.
What is it?
On 20 October, ElevenLabs released Eleven Realtime: two models, Mini and Max. They manage speech recognition, the moment you finish speaking, the thinking process, and the voice in one package. A caveat: the documentation states that under the hood, it is still a chain of components. ElevenLabs simply tunes that chain for you.
Mini is intended for regular conversations at low cost. It understands when you've finished speaking, recovers if you interrupt, and doesn't stop when you say “hm-hm.” Max is for more demanding tasks, such as customer service.
Both work in over 90 languages, with a choice of more than 17,000 voices.
How Good is it?
Max starts speaking on average after 0.57 seconds and correctly answered 98.8 percent of spoken reasoning questions. Mini scored 75.9 percent in that regard. Notably, Mini performs slightly better in the natural back-and-forth of a conversation, 95.8 versus 94.1 percent. Thus, Mini feels more human, while Max is smarter.
A small note: the figures come from tests by Artificial Analysis, but as reported by ElevenLabs themselves.
Who is it for?
For developers, not consumers. It is included in ElevenAgents and a new Realtime API. Those already working with OpenAI’s speech API can switch with two lines of code, according to ElevenLabs. This is a clear challenge to OpenAI.
Be mindful of privacy. Only Max has a mode where nothing is stored, and that only in the US and the EU. Mini does not have this mode at all.
Why This Interests Us
We are developing our own local voice assistant with Lone, and I am busy creating an AI chat website (more on that later). We are encountering precisely this problem: how to get a response quickly enough that it feels like a conversation? ElevenLabs solves it in the cloud. We do it locally, on our own hardware. Two routes to the same goal. It will be interesting to see which route prevails.
