It’s Tokens All The Way Down: How Recursive Language…

Kevin Madura explains recursive language models that treat context as an executable object, boosting reasoning accuracy from 2.6% to 45.4% and reshaping how AI handles complex tasks.

Ever wonder why a language model can’t just add up dozens of numbers scattered across a massive context? Kevin Madura shows a radical shift that turns context into a live REPL.

⚡ TL;DR: RLM treats context as an executable object, not just tokens. It can slice, compute, delegate to sub‑models, and keep logic, execution, and results together. This boosts accuracy dramatically on complex reasoning tasks.

The Simple Version

Think of a language model as a notebook that can run code on the fly. Instead of rereading a huge document, it treats the whole thing as a variable you can manipulate.

How It Actually Works

Context as an Object

🎯 Think of it this way: It’s like handing a chef a pantry and letting them pull out exactly the ingredients they need, instead of reading the entire grocery list.

RLMs expose the context as a mutable object inside a REPL, so the model can slice, aggregate, or transform it directly.

Recursive Delegation

🎯 Think of it this way: Imagine a detective calling in a specialist for each clue, then handing back only the solved piece.

The model can spawn sub‑models with their own parameters, solve a sub‑problem, and return just the result to the parent context.

Deterministic Shell

🎯 Think of it this way: It’s a stable sandbox where the model fills in the middle of a script, like a calculator that writes its own code.

During inference the model generates code that runs in a controlled environment, ensuring reproducibility.

Technical Deep Dive

RLMs replace token‑wise attention with a REPL‑style interface. The input is parsed into a data structure (often a Python dict or pandas DataFrame) that lives in memory. The model then executes Python snippets, evaluates expressions, and feeds the output back as part of the next prompt. Because execution happens inside the same process, there is no need for external APIs or JSON shuffling. This design eliminates “context rot” – the degradation that occurs when long contexts are compressed or truncated. The deterministic shell also allows precise timing: the model decides when to stop reasoning and output a final answer.

Why This Matters

Traditional RAG pipelines stuff thousands of tokens into a window, causing quality loss as the model struggles to prioritize relevant bits. Agents built on tool calls often lose coherence because logic, execution, and results travel across separate JSON messages. RLMs unify these stages, keeping reasoning, computation, and output in a single environment. This leads to measurable gains – a recent benchmark lifted accuracy from 2.6 % to 45.4 % on complex reasoning, especially when the problem reduces cleanly to code. As models are post‑trained to be RLM‑aware, we can expect even more sophisticated autonomous workflows.

Key Takeaways

RLMs view context as an executable object, not just a bag of tokens.
They can delegate sub‑problems to internal models, enabling recursive decomposition.
Logic, execution, and results stay together, preventing the “context rot” seen in RAG.
Benchmarks show dramatic accuracy jumps on tasks that map to code execution.
The deterministic REPL shell makes reasoning reproducible and debuggable.
Potential applications include long‑document analysis, log pattern mining, and codebase‑wide queries.
Future models may be pre‑trained to be RLM‑ready, unlocking richer autonomous behavior.

It’s Tokens All The Way Down: How Recursive Language Models Are Changing AI Reasoning