When an AI companion's response time drifts from two seconds to two and a half, it doesn't just feel slow—it feels broken.
For the team at Tolan, that half-second delay was enough to tank every single product metric and alienate their user base.
In this deep dive, we look at how engineer Paula Dozza and her team re-engineered their entire pipeline to handle the chaotic, unpredictable reality of human speech.
The Simple Version
Most people build AI apps thinking about text boxes, where you type, hit enter, and wait for a reply.
But voice is a completely different beast; it is fast, messy, and full of interruptions.
Imagine you are talking to a friend while cooking dinner; you might stop mid-sentence, walk away, or suddenly interrupt them with a new thought.
Tolan's engineering philosophy is that you cannot use "text rules" for a "voice world."
How It Actually Works
The Latency Trap
In a text-based LLM application, a few seconds of latency is often acceptable because users are used to the "thinking" rhythm of reading.
In voice, however, humans expect near-instantaneous feedback to maintain a natural conversational flow.
Tolan discovered that even a 500-millisecond increase in latency fundamentally changed how users perceived the AI's personality.
Smart Turn Detection
One of the biggest headaches in voice AI is knowing when a user has actually finished speaking.
If the AI cuts you off too early, it feels rude; if it waits too long, it feels robotic and slow.
Instead of just looking for silence, Tolan built turn detection that analyzes speech patterns to distinguish between a meaningful pause and a mid-sentence breath.
They actually decided to "pay" 60 milliseconds of latency to implement better detection, which ended up halving the most frustrating early cut-offs.
Technical Deep Dive
Granular Pipeline Measurement
To fix their speed issues, the team stopped looking at "total latency" as a single number.
Knowing a response took 2.5 seconds is useless if you don't know where those seconds went.
They began measuring every single stage of the pipeline separately—from audio ingestion and STT (Speech-to-Text) to LLM inference and TTS (Text-to-Speech) generation.
Routing by Emotional Stakes
Not every sentence in a conversation requires a massive, expensive model like GPT-4o or Claude 3.5 Sonnet.
Tolan uses a cheap classifier to read the "emotional stakes" of a turn before routing it.
If it is a user's very first conversation or a high-emotion moment, the system routes to the strongest model available.
If it is just idle chatter, the turn is routed to a much smaller, faster model, which handles about a third of all turns without any measurable drop in user retention.
Memory as a Retrieval System
Most LLM apps treat memory as a long transcript of everything said, which quickly becomes bloated and confusing.
Tolan treats memory as a retrieval system that is compressed nightly.
During this nightly process, the system merges duplicate facts and resolves contradictions to ensure the AI doesn't remember two different versions of the same story.
Dynamic Context Reassembly
In text apps, developers often use "warm caches" to keep the previous context ready for the next turn to save time.
Tolan realized this is a trap for voice; if a user suddenly changes the subject, a warm cache makes the AI "confidently wrong" by trying to link the new topic to the old one.
Instead, they reassemble context from parts every single turn, ensuring the model is always grounded in the immediate, volatile reality of the current conversation.
Why This Matters
Tolan's approach proves that voice-first AI is not just an "add-on" feature for LLMs; it is a fundamental architectural shift.
As we move toward more agentic, embodied AI, the ability to handle real-time, messy human interaction will be the primary differentiator between a novelty tool and a true companion.
The lessons learned from managing 4 million hours of spoken data provide a roadmap for anyone building in the high-stakes world of real-time interaction.