Imagine showing an AI a photo of half a chessboard and asking how many white squares are visible. A human counts the squares in the frame; the AI looks at the image, recognizes it's a chessboard, and confidently answers "32" because that's how many squares are on a full board.
This isn't intelligence—it's a high-speed guess based on a memory of patterns. It is the core reason why current frontier models, despite their brilliance in text, still reason like toddlers when it comes to the physical world.
The Simple Version
Most AI models today don't actually "see" the world; they recognize labels. If you show a model a picture of a Golden Retriever, it doesn't analyze the shape of the ears or the texture of the fur in real-time—it matches the image to a massive database of "dog-like" patterns it saw during training.
This works great for naming flowers or identifying breeds, but it fails the moment you ask a question that requires spatial reasoning (like counting specific objects in a cluttered room) or temporal consistency (like tracking a robot arm moving a lid over several seconds).
How It Actually Works
Pattern Matching vs. Spatial Grounding
Andrew Dai points out that the very thing that makes models great at identification—pattern matching—is exactly what causes them to fail at reasoning. As Dai puts it, "the pattern matching is actively hurting them in this case."
When a model sees a partial image of a board game like Catan, it doesn't count the roads on the board. Instead, it might see pieces sitting off to the side and hallucinate a total based on what a "standard" game setup looks like, rather than what is actually visible in the pixels.
The "One Second Test"
To distinguish between visual understanding and visual reasoning, Dai proposes a simple heuristic: the one-second test. If a human can answer a question about an image in under one second (e.g., "What flower is this?"), it's a pattern recognition task.
If the question requires more than a second to process (e.g., "How many red hotels are in this photo?"), it is a "System 2" thinking problem. This requires the model to actively scan, hold state, and reason through the image—areas where frontier models currently fall apart.
The Benchmark Illusion
We often hear that models are "solving" visual benchmarks, but Dai argues these metrics are misleading. He cites ARC-AGI, noting that many of its images are only 32x32 or 64x64 pixels. In the real world, almost no complex task can be reduced to that resolution.
Similarly, he mentions MMU (Massive Multidiscipline Multimodal Understanding), noting that many of its science questions can be answered using text-based knowledge alone, without the model ever needing to actually "reason" through the provided image.
Technical Deep Dive
To understand why this happens, we have to look at how multimodal models are typically built. Most use a Vision Transformer (ViT) or a similar encoder (like CLIP) that breaks an image into patches. These patches are then flattened into tokens and fed into a Large Language Model (LLM).
The problem is that the LLM treats these visual tokens much like words. It searches for statistical correlations between the visual tokens and the text output. It isn't building a 3D geometric map of the scene; it's predicting the most likely string of text to follow a specific set of visual patterns.
Elorian's Four-Stage Approach
To move beyond this, Elorian is implementing a specific architectural shift. First, they are building a synthetic data flywheel. Since high-quality visual reasoning data doesn't exist in the wild (most internet images are just labeled, not "reasoned"), they use a loop of agents, SFT (Supervised Fine-Tuning), and RL (Reinforcement Learning) to create it.
Second, they are introducing Visual Chain of Thought. In standard LLMs, Chain of Thought (CoT) is a text-based sequence of "Let's think step-by-step." Elorian's version happens natively in the visual space.
Architectural Evolution
Beyond the transformer, this requires the model to maintain a more robust internal state. Dai mentions "context amnesia," where models lose track of events in a video. This is often due to the limited context window of visual tokens—videos generate a massive amount of data that quickly exceeds the model's memory, causing it to forget that a robot arm already lifted a lid three seconds ago.
Why This Matters
This isn't just about counting squares on a chessboard; it's the primary bottleneck for Physical AI. In robotics, a model that cannot maintain spatial consistency cannot perform complex, real-time actions in dynamic environments. Current methods are often "overengineered and brittle" because they rely on static images rather than active physical interaction.
In high-stakes industries like construction, the gap is even more dangerous. Construction sites have complex safety rules (like OSHA policies) written in text. A model that can't match that text to the actual spatial relationship of workers and equipment on a live camera feed cannot effectively enforce safety.
Finally, in mechanical design, the stakes are measured in thousands of hours. Dai notes that designing a small part of a robot testing platform can take 100 to 200 hours, with entire platforms taking up to 3,000 human hours. Because frontier models fail at interpreting 3D CAD and blueprints, they remain useless for the very tasks—like designing rockets or batteries—that require precise geometric logic.