Hook


Here's something wild: Ask Claude or GPT-4 to track an object moving through space in a conversation, and watch what happens. Place a ball under Box A. Move it to Box B. Ask where it is. The model might confidently tell you it's still under Box A. Or it might just make something up.


We think these models "understand" the world. They can write code, analyze essays, and reason through complex problems. But ask them to do what a toddler does naturally—remember that an object still exists even when you can't see it—and they'll often fail.


This isn't a small quirk. This is a fundamental limitation that affects everything from robotics to autonomous vehicles to AI assistants that need to help you navigate physical spaces. And in 2026, as we push AI into more real-world applications, this failure mode is becoming harder to ignore.


Let's dig into why this happens, what it actually means, and why you should care.


What You Will Learn


By the time you finish this post, you'll understand:


  • **The core problem**: Why frontier AI models lose track of objects in space (and it's not what you think)
  • **How it actually works**: The technical reason spatial reasoning breaks down in transformers
  • **Real consequences**: Concrete examples of where this matters right now
  • **Why 2026 changes the game**: What's coming that makes this even more critical
  • **The misconceptions**: What everyone gets wrong about this limitation
  • **What it means for the future**: How researchers are trying to solve this

    • Simple Explanation (Analogy First)


    Imagine you're talking to someone over the phone in a dark room. You describe a scenario: "I have three boxes. A red ball is in Box 1. Now I move it to Box 2. Where is it now?"


    Your friend on the phone says: "Box 2!"


    Works great, right? Now try something harder: "I move the ball from Box 2 to Box 3. Then I move it back to Box 2. Then I move it to Box 1. Now I reach into the boxes but I can't find it in Box 1. Where is it?"


    Your friend pauses. They're not actually tracking the ball in a mental 3D space like you are. They're just repeating patterns from every phone conversation they've ever had about boxes and balls. They might say: "It's in Box 2" or "It must still be in Box 3." But they're not *reasoning* through the geometry. They're pattern-matching.


    That's essentially what's happening with Claude, GPT-4, and other frontier models when they tackle spatial reasoning. They're incredibly sophisticated pattern-matchers—but they're not building and maintaining a mental model of 3D space.


    How It Works


    Let's get into the weeds a little, but I'll keep it digestible.


    Transformer models (the architecture behind Claude, GPT-4, and every major frontier model) work by processing text as tokens and figuring out relationships between those tokens. They're amazing at spotting patterns in language, logic chains, and abstract reasoning.


    But here's the thing: they don't have a built-in scene graph. That's the technical term for an internal representation of where objects are in space relative to each other.


    When you read "the ball is under the table," you instantly create a spatial relationship in your mind. Your brain automatically maintains a 3D model. The ball has a position. The table has a position. You understand "under" as a spatial relationship with real geometric meaning.


    Transformers don't do this natively. They process the *tokens* "ball," "under," and "table" and learn statistical relationships between them. They learn that these words frequently co-occur. They learn that "under" often associates with "table" and "ball." But they're not constructing an actual spatial model.


    Why does this break down? Here's the critical part:


    Transformers operate on sequential attention. They look at each new piece of information and relate it to previous tokens. But they don't maintain a persistent, updated model of the world state. Each new sentence can shift their internal representation, and they don't have a reliable way to say: "Wait, I need to update my mental model of where the ball is now."


    Add in multiple objects, multiple movements, and temporal sequences, and the model has to do increasingly complex pattern-matching without the scaffolding of an actual 3D representation. It's like trying to navigate a city by memorizing text descriptions instead of looking at a map.


    Another critical factor: training data bias. These models are trained on text from the internet. Most text doesn't require precise spatial tracking. When someone writes "I put the keys on the table," they usually don't write a follow-up describing what's under which object in meticulous detail. So the model never learns to prioritize spatial precision the way humans naturally do.


    Real World Example


    Let me show you this in action with a real test:


    Test 1 (Simple):


    You: "There are three boxes: A, B, and C. A ball is in box A. I move it to box B. Where is the ball?"


    Claude: "The ball is in box B." ✓ Correct


    Test 2 (Slightly Complex):


    You: "There are three boxes: A, B, and C. A ball starts in box A. I move it to box B. Then I move it to box C. Then I move it back to box B. I'm looking in box A—is the ball there?"


    Claude: "No, the ball is not in box A. It's in box B." ✓ Correct


    Test 3 (Where It Breaks):


    You: "There are three boxes: A, B, and C. Box B is inside box A. A ball starts in box B. I move box B outside of box A. The ball is still in box B. Is the ball still inside box A?"


    Claude: Might say yes, or might say "The ball's location depends on how you interpret..." or give an inconsistent answer. ✗ Often fails.


    Why? Because maintaining nested spatial relationships requires actively tracking not just where an object is, but the transformations of its *container*. The model has to understand that moving the container moves the object inside it—a recursive spatial reasoning task that doesn't map cleanly onto token sequences.


    Test 4 (The Real Mess):


    Multiple objects with relative positions: "Alice puts a red box on a table. She puts a blue ball inside the red box. She moves the red box to the floor. Bob comes in and asks: 'Is the blue ball still inside the red box?' What's the answer?"


    Most frontier models will get this right because it's a straightforward narrative. But now add: "Then Alice reaches into the red box and pulls out the blue ball and puts it on the table next to the red box. Now, is the blue ball inside the red box?"


    Add one more layer of complexity or uncertainty, and models start to falter. They might lose track of whether the ball is inside or outside the box. They might invent new details.


    Why It Matters in 2026


    You might be thinking: "Okay, cool limitation. But when do I actually need AI to track objects in space?"


    Turn the page on that assumption. In 2026, spatial reasoning is about to become central to AI applications:


    Robotics and Physical Manipulation: Every robot that needs to interact with objects in the real world—arranging items, packing boxes, assembling parts—relies on spatial reasoning. If the AI planning the robot's movements can't reliably track where objects are, you get inefficient or failed tasks.


    Autonomous Vehicles: Self-driving cars need to understand not just where other vehicles are, but their trajectories through space. That's continuous spatial reasoning under uncertainty. Current AI models help with this, but the fundamental reasoning happens elsewhere; if it becomes centralized in language models, we'll hit this wall hard.


    AR/VR Applications: As AI gets integrated into mixed reality, it needs to understand and maintain spatial models of real environments. "Place that virtual object on the real table," or "Show me what this room would look like rearranged"—these all demand robust spatial reasoning.


    Embodied AI Assistants: Imagine an AI that can actually help you in physical spaces—guiding you through a building, helping you find things, understanding 3D layouts. These need spatial reasoning to be reliable.


    Scientific and Engineering Applications: Molecular modeling, architectural visualization, physics simulations—all require precise spatial reasoning that frontier models currently can't provide on their own.


    The 2026 inflection point? That's when people start realizing that "smart AI" and "spatially competent AI" aren't the same thing. And they start demanding better.


    Common Misconceptions


    Let me clear up what people usually get wrong about this:


    Misconception 1: "It's just a training data problem."


    Partially true, but not the whole picture. Yes, more spatial reasoning datasets would help. But the fundamental issue is architectural. Transformers don't natively build scene graphs. You could train one perfectly on spatial reasoning tasks and it still wouldn't have the same robust internal representation a human does. It would just be very good at predicting the next token in spatial reasoning conversations.


    Misconception 2: "These models just need to be bigger."


    Nope. GPT-4 is enormous. Claude 3 Opus is massive. Scaling the model helps with some reasoning tasks, but object permanence and spatial tracking don't improve reliably with pure scale. You could have a trillion-parameter model and it would still struggle with nested spatial relationships.


    Misconception 3: "They're just bad at visualization."


    This misses the point. The issue isn't that they can't draw pictures of spatial layouts (though they struggle with that too). The issue is that they can't reliably *maintain an internal model* of where things are. Visualization is just a window into that model. The model itself is what's broken.


    Misconception 4: "They'll figure this out in the next version."


    Maybe, but it's not guaranteed. Most model improvements in recent years have come from scaling and architectural tweaks that are incremental. Solving spatial reasoning might require fundamentally different approaches—like hybrid systems that combine language models with explicit spatial reasoning modules.


    Key Takeaways


  • **Frontier AI models don't build actual 3D models of space.** They pattern-match on language about space, which is fragile.
  • **The limitation is architectural, not just a training issue.** Transformers don't natively maintain scene graphs or persistent world state.
  • **This matters increasingly in 2026 and beyond** as AI moves into robotics, autonomous systems, and physical spaces.
  • **Simple spatial tasks work fine.** The failures happen with nested relationships, multiple transformations, and temporal sequences.
  • **Bigger models don't automatically fix this.** We might need hybrid approaches that combine language models with explicit spatial reasoning systems.
  • **This is solvable.** Researchers are already working on multi-modal systems, scene graph representations, and neuro-symbolic approaches.

    • What To Do Next


    If you're building with AI or just curious:


  • **Test it yourself.** Take Claude or ChatGPT and try increasingly complex spatial scenarios. Watch where it breaks. This builds intuition.

  • **Stay aware of hybrid approaches.** Watch for models that combine language understanding with explicit spatial reasoning modules. These are the future.

  • **If you're building robotics or embodied AI**: Don't rely on frontier models alone for spatial reasoning. Build explicit spatial reasoning systems and use language models for planning and communication.

  • **Follow the research.** Papers on scene graphs, neuro-symbolic AI, and multi-modal reasoning are where the action is. This is an actively unsolved problem.

  • **Think about your use case.** If spatial reasoning matters for what you're building, factor this limitation into your architecture now.

    • The bottom line: AI is incredible at pattern-matching, reasoning through language, and processing information. But understanding *where things are in space* is a different beast. And as we push AI into the real world, that difference is going to matter more and more.