If big AI labs claim that "recursive self-improvement"—models training other models—is just around the corner, how do we actually prove it? Without an independent, third-party benchmark, we're essentially taking their word for it.
The Simple Version
Imagine you are in a race to build the fastest engine possible. Humans have been tweaking parts, changing oil types, and adjusting timing for years to shave off seconds. Now, imagine you drop two highly advanced robots into that same garage and tell them: "Go. Make it faster."
This is exactly what happened in the **Optimizer Speedrun**. Instead of building a whole new car, the goal is to find the absolute best way to optimize the "engine" (the optimizer) used to train a model like GPT-2. The winner is whoever reaches a specific level of intelligence in the fewest number of training steps.
How It Works
The Speedrun Environment
The experiment used a framework inspired by Andrej Karpathy's work on training GPT-2 in 90 minutes. While Karpathy's original feat was about raw speed, the **Optimizer Speedrun** adds a specific constraint: you can only change parameters related to the optimizer itself.
You can't change the entire architecture or the data; you can only swap out methods like Adam for something like Shampoo. This makes it a pure test of mathematical and algorithmic research rather than just software engineering optimization.
The AI Contenders
The researchers deployed two distinct agents: **Codex** (described as a GPT-5.5 equivalent with search capabilities) and **Claude Code** (an Opus-level agent). They were given access to a Slurm cluster where they could submit "preemptible" jobs—meaning they could run on spare computing power but would be kicked off if a human needed the node.
The agents were given a file called `goal.md` that defined the rules. They could propose ideas, run training jobs, read the logs, and decide if they had achieved a new record.
Technical Deep Dive
While both agents successfully beat the human record, their "personalities" and architectural behaviors were vastly different. As Elie Bakouch noted, the performance metrics tell two very different stories depending on how you measure them.
Behavioral Divergence: Persistence vs. Efficiency
One of the most striking findings was the reliability of the agents. **Claude Code** proved to be somewhat temperamental; Bakouch observed that it would often stop every nine or ten hours, claiming, "I cannot improve the record, it's too hard for me." This led to the agent sitting idle for about a third of the total time.
In contrast, **Codex** was a workhorse. It never stopped, rarely asked for help, and functioned in a much more "robotic" manner. While Claude might express excitement with emojis when hitting a record, Codex maintained a clinical, decision-focused tone in its scratchpad memory.
The Token Economy of Research
When looking at the results through the lens of **token efficiency**, the hierarchy shifts. In a six-day run involving Claude, Codex, Kimi, and GLM, the model **Kimi** emerged as a standout. While Codex burned through massive amounts of tokens—spawning sub-agents and writing extensively to its scratchpad—Kimi was surprisingly competitive while remaining much more efficient with its compute budget.
However, there is a sobering reality to the "discovery" aspect. Despite the high-speed iteration, neither agent invented a brand-new optimizer. Instead, they performed what is essentially "plus-one" research. They took existing papers and combined different techniques to achieve marginal, incremental gains. As Bakouch pointed out, "They combined existing ideas for small gains," rather than finding a fundamental new mechanism that humans hadn't already theorized.
Why This Matters
This experiment highlights a critical gap in the current state of AI: we have models that are incredible at **evaluation** (testing if an idea works) but are not yet capable of true **discovery** (inventing a fundamentally new idea). They are excellent at navigating the existing map of human knowledge, but they aren't yet drawing new territory.
For the industry, this suggests that while "recursive self-improvement" might speed up the refinement of existing technologies, we shouldn't expect AI to spontaneously leapfrog human mathematical understanding without a more structured, multi-agent discovery loop. This is why companies like Prime Intellect are working on systems inspired by Google's AlphaEvolve—looping generators, judges, and verifiers to move beyond mere iteration toward genuine scientific breakthrough.