KV Cache-Aware Routing & Disaggregation: Supercharging LLM…

Discover how KV‑cache routing and pod‑level disaggregation cut latency 10× and boost throughput on Kubernetes, enabling cheaper, faster LLM serving without extra GPUs.

Ever wonder why some AI chats feel instant while others lag for seconds? This talk reveals the hidden tricks behind lightning‑fast LLM responses on Kubernetes.

⚡ TL;DR: KV‑cache routing and pod‑level disaggregation cut latency by 10× and boost request throughput, letting you serve longer prompts without extra GPUs.

The Simple Version

When a client owns the KV cache, subsequent turns can reuse cached tokens instead of recomputing them. That reuse creates a huge speed gap between cached and uncached inference.

🎯 Think of it this way: It’s like rereading a familiar chapter instead of rewriting the whole book.

How It Actually Works

Endpoint Picker

llm-d watches cache hit ratios and selects the pod that currently holds the warmest cache. The picker then routes the request there.

🎯 Think of it this way: Choosing the fastest checkout line at the grocery store.

Offload Tiers

If a pod runs out of memory, requests are shifted to a larger tier with more RAM, keeping the overall service level high.

Eviction Strategy

Cold caches are dropped based on a TTL, ensuring hot ones stay alive for the next request.

Technical Deep Dive

llm-d implements a lightweight HTTP router in Go that reads the X‑Cache‑Hit header from each pod. It uses a consistent hash on the request ID to keep sticky sessions, and falls back to a fallback pool when no cache is warm.

The routing decision runs in under 100 µs, allowing the system to keep up with bursty traffic.

Disaggregation splits prefill (compute‑heavy) from decode (memory‑heavy) by assigning separate workers to each phase. Cache can be moved between workers over RDMA or RoCE, which reduces the need for duplicate GPU memory.

Why This Matters

Real‑world deployments see a 10× reduction in per‑token cost when routing leverages cached states. That means you can serve longer prompts or handle more concurrent users without adding GPUs.

Lower latency also improves user experience for multi‑turn agentic workflows, where each turn depends on the previous context.

By decoupling cache movement from the compute plane, teams can scale horizontally and avoid the “GPU wall” that traditionally caps inference capacity.

Key Takeaways

Cache reuse can slash token cost by up to 10×.
Routing based on cache heat is cheaper than adding more GPUs.
Disaggregation separates prefill and decode workloads for better resource utilization.
RDMA or RoCE fabrics are required to move cache between workers efficiently.
Pod‑affinity rules keep subsequent turns on the same pod for minimal latency.
TTL‑based eviction keeps the hottest caches alive while freeing stale memory.
Benchmark results show P99 latency dropping from ~900 ms to ~100 ms under the new setup.

KV Cache-Aware Routing & Disaggregation: Supercharging LLM Inference on Kubernetes