Spatial Competence Benchmark
| Authors | Jash Vira & Ashley Harris |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.09594 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Spatial competence is the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints. Prevailing spatial evaluations for large models are limited to probing isolated primitives through 3D transformations or visual question answering. We introduce the Spatial Competence Benchmark (SCBench), spanning three hierarchical capability buckets whose tasks require executable outputs verified by deterministic checkers or simulator-based evaluators. On SCBench, three frontier models exhibit monotonically decreasing accuracy up the capability ladder. Sweeping output-token caps shows that accuracy gains concentrate at low budgets and saturate quickly, and failures are dominated by locally plausible geometry that breaks global constraints. We release the task generators, verifiers, and visualisation tooling.
Engineering Breakdown
Plain English
This paper introduces SCBench, a benchmark designed to measure how well large language models understand and reason about spatial environments. The benchmark organizes tasks into three difficulty levels that require models to produce executable outputs (like code or structured actions) that are verified by deterministic checkers or physics simulators, not just text answers. When tested on frontier models like GPT-4, Claude, and Gemini, accuracy drops monotonically as tasks get harder, and the authors discovered that output tokens are most valuable at low budgets before quickly saturating—most failures come from locally plausible but globally incorrect geometry rather than complete breakdowns.
Core Technical Contribution
The core novelty is a hierarchical evaluation framework that moves beyond isolated 3D transformation proofs or visual question answering (VQA) by requiring models to produce executable, verifiable outputs under resource constraints. Rather than asking "does the model understand rotation?" in isolation, SCBench asks "can the model generate correct code or actions that physically work in a constrained environment?" The benchmark introduces the key insight that output-token budgets fundamentally shape spatial reasoning accuracy—models spend tokens inefficiently early on and hit diminishing returns quickly. This is the first systematic study showing that spatial reasoning saturates rapidly in frontier models, suggesting a bottleneck in how these models represent and plan with spatial information.
How It Works
The benchmark operates in three hierarchical tiers: tier 1 tests basic spatial primitives (single object transformations, distance reasoning), tier 2 combines primitives (multi-object scenes, constraint satisfaction), and tier 3 requires full planning under hard constraints (navigating complex environments, generating executable paths). For each task, the model produces outputs—either code snippets, coordinate sequences, or action plans—that are executed against ground truth or passed to a physics simulator (likely PyBullet or similar) to verify correctness. The evaluation uses deterministic checkers for symbolic outputs ("did the generated path avoid obstacles?") and simulator-based metrics for physical plausibility ("does the trajectory respect collision and dynamics constraints?"). The authors sweep output-token limits from low (5-10 tokens) to high (100+ tokens) budgets and measure where accuracy gains plateau, revealing that most value concentrates at low token budgets and failures cluster around locally valid but globally inconsistent geometry.
Production Impact
For engineers building spatial AI systems (robotics planning, 3D scene understanding, embodied agents), this benchmark provides a standardized way to diagnose whether your model actually understands spatial constraints or just pattern-matches plausible answers. If you're deploying an LLM to generate code for path planning or object manipulation, SCBench reveals you likely need aggressive output-token budgeting—allocating 50+ tokens per spatial query may waste compute on diminishing returns, while 10-20 tokens may be the sweet spot. This directly impacts latency and cost: you can optimize inference by capping tokens early and reallocating budget to ensemble or retry strategies instead. The deterministic and simulator-based verification methods are production-ready checkpoints you can integrate into CI/CD pipelines to catch spatial hallucinations before deployment (e.g., a model confidently generating a collision-ridden trajectory). The hierarchical structure also guides curriculum—start by ensuring your models master tier-1 primitives before tackling full planning, avoiding expensive training on impossible tier-3 tasks with unprepared models.
Limitations and When Not to Use This
The paper's scope is limited to evaluating frontier closed-source models (GPT-4, Claude, Gemini) without access to weights, preventing analysis of where spatial reasoning fails in the model's internals—you can see that it fails but not why. The benchmark assumes deterministic or simulator-based verification is always feasible, but real-world spatial tasks (e.g., manipulation in uncontrolled lighting) have ambiguous ground truth that simulations can't fully capture. The paper doesn't address how spatial reasoning scales with scene complexity—all tested tasks likely stay within modest scene graphs or coordinate spaces, so it's unclear if conclusions hold for large photorealistic environments or high-dimensional constraint spaces. The monotonic degradation result (accuracy drops tier-by-tier) is observed but not deeply analyzed: it's unclear whether the issue is fundamental (spatial reasoning requires exponential compute) or solvable (the models just need different training).
Research Context
This work builds on a decade of spatial reasoning evaluations in vision (CLEVR, 3D-FRONT) and embodied AI (Habitat, Replica), but SCBench is the first to systematically benchmark spatial competence in LLMs with executable verification. It directly responds to the limitation that VQA and 3D transformation proofs test isolated skills without requiring coherent scene models or action planning. The benchmark parallels recent work on code generation verification (HumanEval-style checkers) but applies it to spatial domains, establishing a methodological template for executable AI evaluation. The token-budget sweep and saturation findings open a new research direction: understanding whether spatial reasoning in LLMs is fundamentally limited by architecture (transformers may struggle with recursive spatial constraints) or by training (models need spatial-specific pretraining or RL from spatial feedback).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::