Why is LLM inference harder to scale than traditional web services?

Traditional services are mostly stateless, while LLM inference is stateful due to the KV cache, which must be managed across distributed systems to avoid expensive re-computation.

What is the 'inference control plane'?

It is a logical orchestration layer that manages routing, batching, caching, and scheduling as a single integrated system rather than separate, disconnected components.

What does 'cost per successful task' mean?

It is a metric that accounts for the total resources spent (compute, retries, failures) to complete an entire user goal, rather than just measuring the cost of generating individual words (tokens).

Scaling AI Inference: Why Orchestration is the New Frontier

Meta engineers Nishant Gupta and Naman Ahuja explain why AI inference is moving from simple model serving to complex, distributed orchestration at scale.

Running large language models (LLMs) at the scale of Meta isn't just about having the biggest GPUs; it's about managing a chaotic, distributed web of moving parts. As AI moves from research experiments to foundational infrastructure, the complexity is exploding faster than any workload in history.

⚡ TL;DR: Meta engineers Nishant Gupta and Naman Ahuja reveal that AI inference is transitioning from simple model serving to a complex orchestration phase. To scale effectively, companies must move beyond optimizing "tokens per second" and instead build a dedicated inference control plane that manages GPU memory, KV cache state, and routing as a unified distributed system.

The Simple Version

Imagine you are running a massive global pizza delivery empire. In the early days, you just needed a good chef (the model) and an oven (the GPU). If more people ordered, you just bought more ovens.

But as you grow into a global giant, the problem changes. Now, you have to worry about which driver is closest, whether the dough is already pre-made (caching), how to group multiple orders into one delivery run to save gas (batching), and what to do if a driver gets a flat tire halfway through a delivery (reliability).

In the AI world, the "chef" is the model, but the "delivery empire" is the orchestration layer. As Nishant Gupta explains, we are currently seeing the orchestration layer emerge in real time to handle the massive complexity of modern AI workloads.

How It Actually Works

To understand why AI inference is so much harder than traditional web services, we have to look at the fundamental shift in how requests work.

From Stateless to Stateful

Traditional microservices are mostly stateless. If a server crashes, you just spin up another one and keep going. LLM inference, however, is deeply stateful because of the KV cache.

The KV cache stores the mathematical context of a conversation so the model doesn't have to re-process everything from scratch every time a new token is generated. As the speakers note, "the KV cache... is very expensive to build and even more expensive to throw away." This makes every request feel less like a simple function call and more like a distributed transaction.

🎯 Think of it this way: A traditional web request is like asking a librarian for a fact—they give it to you, and you leave. An LLM request is like a long, collaborative research project where the librarian has to remember everything you've discussed so far to make sense of your next question.

The Seven Axes of Scheduling

In the old days of cloud computing, a scheduler only had to worry about a few things, like CPU and RAM. For AI, a scheduler has to juggle at least seven different dimensions simultaneously:

GPU Generation: Managing a mix of hardware like H100s, H200s, or B200s.
Memory Headroom: Ensuring there is enough VRAM available.
KV Cache State: Knowing if the context is already sitting in a specific GPU's memory.
Weight Warmth: Whether the model weights are already loaded or require a slow "cold start."
Tenant Priority: Handling different users with different service level agreements (SLOs).
Latency Budget: How much time is left before the user experience breaks.
Workflow Context: Knowing if a request is the first step of a task or the final step of a long, expensive reasoning chain.

Technical Deep Dive

The most profound technical insight from Meta's engineers is the concept of tight coupling. In traditional systems, layers like routing, caching, and autoscaling are relatively independent. In AI inference, they are inextricably linked.

A single decision at the routing layer—such as sending a request to a specific region—can change the cache hit rate. This change in hit rate alters the batch composition (how many requests are grouped together), which in turn changes GPU utilization, which ultimately triggers a different response from the autoscaler. Everything is entangled.

The Four Quadrants of Optimization

To manage this complexity, Gupta proposes a framework for optimization that categorizes every technique into one of four quadrants:

Avoid the work: Using prefix caching or semantic caching to skip computation entirely.
Share the work: Using continuous batching or speculative decoding to make better use of compute cycles.
Move the work: Routing requests to a smaller, cheaper model or a different geographic region.
Delay the work: Using admission control to queue requests based on priority and deadlines.

This framework highlights why the industry is moving toward an inference control plane. Much like Kubernetes emerged to manage the complexity of virtual machines, an inference control plane is needed to treat models, KV caches, and GPUs as unified, schedulable resources.

Why This Matters

For developers and companies building AI agents, the takeaway is clear: the bottleneck is no longer the model; it is the orchestration. As agentic workflows become more common—where a single user prompt might trigger hundreds or even thousands of model calls—the cost and reliability of the underlying infrastructure become the primary drivers of product success.

Meta's engineers argue that we must stop optimizing for the wrong metric. "It's not important to optimize just cost per token or cost per request," Gupta emphasizes. "We have to optimize cost per successful task." If a system is fast and cheap but fails halfway through a complex reasoning chain, it has provided zero value to the user.

As we move into the era of autonomous agents, the winners won't just be those with the smartest models, but those with the most sophisticated control planes to manage the massive, stateful, and highly coupled dance of distributed inference.