What is the difference between half-duplex and full-duplex AI?

Half-duplex AI can only do one thing at a time: either listen or speak. Full-duplex AI can do both simultaneously, allowing for natural interruptions and overlapping conversation.

Why can't LLMs just process raw audio files?

Raw audio has too many data points (timesteps). Because attention costs grow quadratically, processing raw audio would be too computationally expensive and slow.

What are neural codecs in the context of voice AI?

Neural codecs are compression tools that turn complex audio waves into discrete tokens that a language model can process efficiently, similar to how it processes text.

Gradium and the Quest for Full Duplex Voice AI

Learn why current voice AI feels like a walkie-talkie and how Gradium is working to enable 'full duplex' conversations with simultaneous speaking and listening.

Have you ever noticed that talking to a voice AI feels less like a conversation and more like using a walkie-talkie? You speak, you stop, you wait for the machine to process, and then it responds.

⚡ TL;DR: Neil Zeghidour and Gradium are tackling the 'half-duplex' limitation of current voice AI. While models like GPT-4o have reduced latency, they still can't handle simultaneous talking (full duplex). Gradium aims to solve this by modeling two token streams at once using neural codecs to manage audio complexity.

The Simple Version

Most voice AI today is half-duplex. In simple terms, this means the AI can either listen or speak, but it cannot do both at the same time. If you interrupt it or make small sounds of agreement, the system often glitches or stops dead.

In real human conversations, we overlap constantly. Research shows that in close relationships, people talk over each other up to 20% of the time. We use 'back-channeling'—those little "mm-hmm" and "yeah" sounds—to show we are listening without taking over the conversation.

🎯 Think of it this way: Current AI is like a walkie-talkie where you have to push a button to talk and release it to hear. True human conversation is like a telephone call where both people can hear and speak simultaneously, reacting to each other in real-time.

How It Actually Works

To understand why this is hard, we have to look at how voice AI evolved. For years, we used cascaded pipelines: a Speech-to-Text (STT) model turned audio into text, an LLM processed the text, and a Text-to-Speech (TTS) model turned it back into audio.

The Latency Trap

Cascaded systems are slow because every step adds delay. They also 'flatten' emotion because the LLM only sees text, losing the nuance of the speaker's tone. While newer speech-to-speech models collapse this into one pipeline to fix latency, they still struggle with turn-taking.

The Back-Channeling Problem

When you say "mm-hmm" while an AI is talking, a half-duplex model interprets this as an interruption. Instead of recognizing it as a sign of engagement, the model typically stops speaking entirely because it thinks it's now the user's turn to lead the conversation.

🎯 Think of it this way: It's like a dance partner who freezes completely every time you accidentally brush their shoulder, rather than just adjusting their step and keeping the rhythm going.

Technical Deep Dive

The primary technical hurdle for voice AI is the sheer volume of data. Raw audio is incredibly dense. For example, just eight words spoken over three seconds at a 24 kHz sample rate results in 72,000 timesteps.

The Attention Bottleneck

In Transformer architectures, the cost of the attention mechanism grows quadratically with the sequence length. Processing 72,000 tokens for a few seconds of audio is astronomically more expensive than processing a few dozen text tokens for the same sentence.

Neural Codecs to the Rescue

To solve this, Gradium and other leaders use neural codecs. These tools compress raw audio into discrete, token-like representations. This allows the model to 'see' audio as a sequence of tokens, similar to how it sees words, making the computation manageable.

Moving to Full Duplex

To achieve true full duplex, Zeghidour explains that we must move from modeling one stream of tokens to two simultaneous token streams. The model needs to track both the user's audio and its own output in parallel, allowing it to distinguish between a meaningful interruption and a simple 'back-channel' sound.

Why This Matters

The gap between 'half-duplex' and 'full-duplex' is the difference between a tool and a companion. When an AI can handle overlaps and subtle audio cues, the interaction stops feeling like a series of commands and starts feeling like a natural human connection.

However, there is a trade-off. Zeghidour notes a tension between naturalness and intelligence. Because models have a fixed 'weight budget,' spending more parameters on the mechanics of hearing and speaking can sometimes come at the expense of complex reasoning capabilities.

The future of voice AI depends on whether we can scale these speech-to-speech models further or if we need to completely split the 'interface' (the hearing/speaking) from the 'intelligence' (the reasoning engine).