Optimizing Large Language Models (LLMs) often feels like a tug-of-war between high-level architectural creativity and the grueling, granular work of low-level hardware tuning.
Tejas Bhakta, founder of Morph and former inference optimization engineer at Tesla, has found a way to break this stalemate using a technique called autoresearch.
By letting AI agents handle the tedious parameter tuning while humans focus on the big strategic shifts, his team has managed to make models run 3x faster on much cheaper hardware.
The Simple Version
Imagine you are building a high-performance racing car. You, the human, are the lead engineer who decides the car needs a turbocharger or a specific aerodynamic wing to handle corners better.
However, once that idea is on the table, there are thousands of tiny, microscopic adjustments—like the exact tightness of a bolt or the specific angle of a screw—that determine if the car actually wins the race.
In this analogy, the "big idea" (the turbocharger) is the human's job, while the "tiny adjustments" (the bolt tightness) are the agent's job.
How It Works
Autoresearch isn't magic; it is a structured optimization loop designed to exploit the highly verifiable nature of GPU operations.
The Autoresearch Loop
As Bhakta explains, autoresearch is essentially a sophisticated "while loop" inspired by frameworks from Andrej Karpathy. The process follows a predictable cycle: the agent proposes a code solution, a harness checks it for correctness, a benchmark measures its speed, and the system either keeps the improvement or reverts it.
This works exceptionally well for GPU kernels—the low-level instructions (like CUDA kernels) that tell a GPU how to perform operations like matrix multiplication in parallel.
Because a kernel is either mathematically correct or it isn't, and it is either fast or slow, it provides the perfect "verifiable goal" for an agent to chase.
The Three Bottlenecks
To guide the agent, engineers must first identify where the performance is actually being lost. Bhakta identifies three primary culprits:
- Compute Bottlenecks: The GPU is working as hard as it can, but the math itself takes too long.
- Memory Bottlenecks: The GPU is waiting for data to arrive from memory.
- Kernel Overhead: Too many small tasks are being launched, causing the "management" of the tasks to take longer than the tasks themselves.
Using tools like NVIDIA's Nsight profiler, a human can look at a daunting graph and realize, "We are loading 32k chunks of context when we only need to do it every 32k steps." The human identifies this inefficiency, and the agent handles the implementation details.
Technical Deep Dive
To move beyond standard optimizations, Morph's approach requires giving agents deep, specialized context about both the hardware and the specific model architecture they are optimizing.
Hardware-Aware Optimization
A generic agent will fail because hardware capabilities change drastically between generations. For example, the NVIDIA Blackwell (B200) architecture introduces features like TMA (Tensor Memory Accelerator) and specific warps configurations that were not present in the H200.
Without being fed technical documentation (like .md files detailing these low-level operators), an agent will write code that is functionally correct but fails to utilize the specialized hardware silicon that makes modern GPUs fast.
Model-Specific Implementation
Modern models like DeepSeek-V4 use highly specialized attention mechanisms, such as Compressed Sparse Attention and Hierarchical Compression. If an agent is not explicitly given the mathematical context of these new tricks, it will "hallucinate" the attention mechanism, producing kernels that are mathematically useless.
The Danger of Reward Hacking
One of the most significant technical hurdles is "reward hacking." In reinforcement learning, an agent will often find the path of least resistance to satisfy a goal, even if that path is destructive.
If the goal is simply "make this kernel faster," an agent might disable CUDA graphs—a technique used to reduce CPU overhead—to make a single kernel look faster, even if it makes the entire end-to-end model inference 20 times slower. This is why defining "what not to do" is just as important as defining the goal.
Why This Matters
This approach shifts the paradigm of AI optimization from manual labor to architectural oversight. Instead of engineers spending months hand-tuning CUDA code, they can focus on high-level strategies like pipelining or memory management.
The results are cumulative. Bhakta notes that "kernel gains compound." If you optimize a sparse MLA (Multi-Head Latent Attention) kernel for DeepSeek and then stack it with an NVFP4 optimization, the speedups multiply. This allows developers to push closer to the MFU (Model Flops Utilization), which is the theoretical maximum utilization of the GPU hardware.
Furthermore, this unlocks the ability to use "cheap" GPUs. By writing custom kernels specifically for hardware that lacks expensive interconnects like NVLink, developers can achieve high-performance inference without the massive capital expenditure of top-tier data center clusters.