In the world of Large Language Models (LLMs), there is a widening gap between running a model in your basement and running one in a professional data center. While the goal for both is efficiency, the strategies used to achieve it are fundamentally different.
The Simple Version
Imagine you are a chef trying to run a kitchen. If you are working in a tiny studio apartment, your biggest problem is space. You have to 'squish' your tools and ingredients just to fit them in the room; once you've managed to fit everything, you focus on making sure the food still tastes good.
Now, imagine you are running a massive industrial catering hall. You have all the space you need, but your biggest problem is speed. You aren't worried about where the pots go; you're worried about how to move the food from the stove to the plate as fast as possible.
As Philip Kiely puts it, local inference is about getting it working and then making it "less dumb." Data center inference is about getting it working and then making it "less slow." The latest trend is to stop treating the model as a finished product and instead use a dedicated training process to optimize how the model behaves during inference.
How It Actually Works
The Quantization Trade-off
Quantization reduces the precision of numbers (weights and caches) to save memory and bandwidth. A recent technique called TurboQuant uses polar coordinates to compress the KV (Key-Value) cache down to 4 bits instead of the traditional 8 bits.
While this effectively doubles the bandwidth when moving the KV cache through system memory, it comes with a heavy price. Kiely notes that the additional computation required during the decode phase "cuts TPS by more than half," making it a poor choice for data centers but a great choice for memory-starved local GPUs.
KV Compaction and 'Still'
The KV cache stores previous tokens so the model doesn't have to re-process the entire prompt every time. However, as context windows reach millions of tokens, this cache grows linearly, consuming massive amounts of VRAM.
Baseten's research team introduced Still, a method for KV compaction. Instead of keeping a deterministic subset of the cache, Still uses a learned bottleneck. It cross-attends a fixed set of learned query vectors against the full KV cache to emit compact keys and values in a single forward pass.
The Evolution of Speculation
Speculative decoding uses a smaller 'drafter' model to guess the next few tokens, which a larger 'target' model then verifies in one go. The field has moved rapidly through several iterations:
- Basic Speculation: Using a smaller model from the same family (often performed poorly).
- Medusa: Adding multiple decoder heads to the target model.
- Eagle 3: Training a 1-billion parameter model specifically on the hidden states of the target model.
- DFlash: A diffusion-based drafter that proposes 8 to 16 tokens at once rather than one by one.
DFlash treats token generation like image generation, iterating over a sequence of tokens. In production, using a single B200 GPU with a Quen 38B model, DFlash showed a more than 3x improvement in acceptance and TPS compared to Eagle.
Technical Deep Dive
The core shift in modern inference is the blurring line between training and inference. Traditionally, a model was trained and then 'frozen' for deployment. Now, techniques like continuous speculator retraining allow the drafter model to be updated on live prompts, leading to a 20% to 2x lift in token acceptance rates.
From an architectural perspective, DFlash's advantage stems from its bidirectional attention. While standard auto-regressive models only look backward, DFlash uses an attention mask that enforces causal consistency while allowing tokens within a block to attend to each other. This allows the drafter to create more coherent sequences that the target model is more likely to accept.
Regarding hardware, the industry is moving toward NVFP4 (Nvidia's 4-bit floating point) and upcoming Ruben systems. In data centers, the focus has shifted away from compressing the KV cache (like TurboQuant) and toward quantizing the weights. This is because the compute cost of decompressing a 4-bit cache during every decode step is often higher than the cost of simply moving an 8-bit cache, provided the hardware bandwidth is sufficient.
Furthermore, disaggregation is becoming critical. This involves separating the prefill phase (processing the initial prompt) from the decode phase (generating tokens). By moving the KV cache across the system or offloading it to CPU memory, engineers can optimize for batch sizes and throughput more effectively than with a monolithic setup.
Why This Matters
The move toward 'training for inference' creates a powerful flywheel effect. Faster inference allows for the collection of more high-quality synthetic data, which is then used to train better models, which in turn enable even faster inference. This cycle accelerates the deployment of massive context windows that were previously computationally impossible.
For developers, this means the choice of hardware now dictates the optimization strategy. If you are deploying on B200s or the upcoming Ruben chips, you should prioritize weight quantization (NVFP4) and diffusion-based speculation (DFlash). If you are building for the 'basement' (local LLMs), memory-saving techniques like TurboQuant remain the gold standard.
Ultimately, the goal is to move toward a world where the 'intelligence' of a model is decoupled from the 'latency' of its delivery. By treating the inference stack as a trainable component rather than a static piece of software, we can achieve near-lossless performance with a fraction of the traditional compute cost.