Inference articles

4 articles about Inference, newest first.

• How OpenAI Routes LLM Inference at Scale — Learn how OpenAI evolved its inference load balancer from a proportional controller to a global optimizer to minimize latency and protect KV cache locality.
• LLM Inference at Scale: How Memory, KV Cache, and Smart Serving Keep AI Apps Running — Explore the hidden cost of KV cache, GPU memory limits, and optimization tricks that prevent LLM inference failures.
• KV Cache-Aware Routing & Disaggregation: Supercharging LLM Inference on Kubernetes — Discover how KV‑cache routing and pod‑level disaggregation cut latency 10× and boost throughput on Kubernetes, enabling cheaper, faster LLM serving without extra GPUs.
• OpenAI's GPT-5.6 Sol Hits 750 Tokens Per Second – Can It Stay Profitable? — OpenAI's GPT-5.6 Sol runs at 750 tokens per second on Cerebras hardware. Explore the speed, cost, and profit challenges behind the new AI milestone.