Imagine spending ten minutes fighting a voice bot that can spell your name letter-by-letter but still can't pronounce it correctly. You've corrected it twice, it's interrupted you three times, and now it's asking you to repeat everything from the beginning.
This isn't just a series of unfortunate bugs; it's a systemic failure of how we build voice AI. Midam Kim, an ML engineer at ServiceNow, argues that we are treating voice agent failures as a random checklist of glitches when they are actually structured linguistic breakdowns.
The Simple Version
Most developers treat a voice bot like a text bot that happens to have a 'voice skin.' They focus on whether the AI understood the words, but they ignore the 'music' of the conversation—the timing, the tone, and the social cues.
Midam Kim suggests we stop looking at these as random errors and start using a map. This map organizes every possible failure into a specific 'cell' so engineers know exactly which part of the system is breaking.
How It Actually Works
The core of Kim's framework is a grid. On one axis, you have the Communication Channels. On the other, you have the Linguistic Levels. This creates eight distinct cells where any voice AI failure can be categorized.
The Two Channels
Every interaction is split into Listening (how the bot perceives the user) and Speaking (how the bot projects itself). A bot might be a genius at listening but a disaster at speaking, or vice versa.
The Four Levels of Failure
Within those channels, failures happen at four distinct depths:
- Sounds: This is the raw acoustic level. In the listening channel, this is a failure of Automatic Speech Recognition (ASR). In the speaking channel, it's a Text-to-Speech (TTS) issue, such as applying English pronunciation rules to a non-English name.
- Words: This is the semantic level. The bot might hear the sounds correctly but fail to map them to the right meaning or intent.
- Interaction: This is the 'dance' of conversation. This includes turn-taking, barge-in (when the bot cuts the user off), and the timing of responses.
- Mental Model: This is the highest level. It's the user's internal understanding of what the bot is capable of and how it works.
Technical Deep Dive
For developers and ML engineers, the most critical insight from Kim's map is interdependence. In many legacy voice architectures, ASR, NLU (Natural Language Understanding), and TTS are treated as modular 'black boxes' that can be optimized in isolation.
Kim argues that this approach is flawed because failures cascade across the grid. For example, a failure at the Speaking-Sound level (mispronouncing a name) directly damages the Listening-Mental Model level. When the user hears the bot mispronounce their name after being corrected, they stop trusting the bot's intelligence.
This trust erosion changes how the user speaks. They might start over-enunciating or speaking in robotic fragments to 'help' the bot. This shift in user behavior then creates new failures at the Listening-Sound level because the ASR was trained on natural speech, not 'bot-speak.'
Furthermore, Kim highlights a fundamental difference between LLM-based chat and voice agents. In a chat interface, the context window is visual and persistent; the history is written on the screen. In voice, the audio signal is ephemeral—it vanishes the moment it is spoken.
Because the audio vanishes, the only 'permanent' record of the conversation is the user's mental model. If the bot fails a turn-taking cue, the user doesn't just see a glitch; they update their mental model to believe the bot is 'broken' or 'stupid,' which fundamentally alters every subsequent turn in the dialogue tree.
Why This Matters
This framework moves voice AI development away from 'whack-a-mole' bug fixing and toward Conversational Engineering. Instead of just trying to increase ASR accuracy by 2%, engineers can identify that their primary churn is happening at the Interaction level.
It also highlights the systemic bias in current voice models. When a bot applies English pronunciation rules to a non-English name, it's not a 'small glitch'—it's a failure of the Speaking-Sound cell that tells the user they are not the intended audience for this technology.
By mapping failures, companies like ServiceNow can build agents that don't just 'understand' words, but understand the social and linguistic structure of a human call. The goal is no longer just a correct transcript, but a stable, positive mental model in the mind of the user.