Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
| Authors | Yakov Pyotr Shkolnikov |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06459 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.
Engineering Breakdown
Plain English
This paper reveals a major bottleneck in vision-language models: their visual encoders learn continuous geometric information (like hand joint angles) that their text pathways cannot express. Using only a 6,000-parameter linear probe on frozen visual features, the authors extract hand joint angles with 6.1 degrees mean absolute error, but the model's best text output achieves only 20.0 degrees — a 3.3x gap. By fine-tuning with LoRA (rank 16, 2,000 images), they close this gap to 6.5 degrees, proving the problem is not what the visual encoder learns, but that the text generation pathway hasn't been trained to articulate it. The paper also discovers that five different encoder architectures (self-supervised, contrastive, hybrid) converge to identical accuracy despite having low representational similarity, suggesting functional convergence matters more than architectural details.
Core Technical Contribution
The core contribution is demonstrating a pathway-training deficit in vision-language models — the visual features contain geometric information that the text decoder is not trained to access or express. Rather than proposing a new architecture, the authors use minimal intervention (linear probes and targeted LoRA fine-tuning) to prove that the bottleneck is training objective, not model capacity or representational power. The finding that five architecturally distinct encoders converge to equivalent functional performance (R² ≈ 0.55) despite low CKA similarity (0.41) suggests a new principle: different internal representations can achieve identical task performance, challenging the assumption that representational convergence is necessary for functional equivalence. This inverts common thinking about what drives model performance — the paper argues optimization landscape and training objectives matter more than architectural choices.
How It Works
The experimental pipeline begins with frozen vision-language model encoders (five variants tested) that extract visual representations from hand images. On top of these frozen features, the authors train a lightweight linear probe (6,000 parameters) with MSE loss to predict hand joint angles directly — this achieves 6.1° MAE, establishing an upper bound on what information exists in the visual features. In parallel, they extract the best text output from the model's language decoder for the same hand images, which achieves only 20.0° MAE. To close the gap, they apply LoRA fine-tuning (rank 16) to the text pathway using only 2,000 labeled images, reducing the text-based error to 6.5°. Finally, they compute representational similarity (CKA) between the five encoder variants and show that despite low similarity scores (0.41), all five converge to R² ≈ 0.55 accuracy on the joint angle task, measured via TOST equivalence testing with delta=0.03.
Production Impact
For teams building vision-language systems, this reveals that geometric or quantitative outputs should not be forced through text decoding — using learned linear heads on frozen representations is 3.3x more accurate for continuous-valued tasks. If you're deploying a vision-language model for pose estimation, robotics control, or any task requiring precise geometric output, bypassing the text pathway and adding task-specific linear layers is a high-ROI intervention (2,000 labeled examples, minimal retraining). The finding that architecture matters less than training objective means you can confidently reuse pre-trained encoders without fine-tuning them — the gains come from training the output head and decoding pathway correctly. However, this trades off multimodal interpretability; you lose the ability to generate natural language explanations alongside numeric predictions, which may be a limitation for some applications requiring explainability.
Limitations and When Not to Use This
The paper focuses exclusively on hand pose estimation — it's unclear whether the 3.3x bottleneck and pathway-training deficit generalize to other geometric tasks (3D object detection, depth prediction, segmentation). The linear probe baseline assumes frozen representations are optimal; in practice, end-to-end fine-tuning of the full encoder might outperform this setup, but the paper doesn't explore that comparison. The TOST equivalence test showing convergence across architectures uses relatively loose delta=0.03 bounds and only measures one task; it's possible that with higher precision requirements or out-of-distribution data, the choice of encoder architecture becomes material. Finally, the paper doesn't address failure modes like occlusion, extreme pose angles, or cross-domain generalization, leaving open whether the learned linear head or LoRA adaptations are robust in production conditions.
Research Context
This work sits at the intersection of vision-language model analysis and task-specific adaptation. It builds on prior work showing that pre-trained encoders contain rich, task-relevant information (e.g., CLIP, BLIP) but extends that observation to reveal a decoding asymmetry — visual features are more informative than text outputs can be. The paper implicitly critiques the trend toward scaling up language models for all tasks; instead, it suggests that for geometric or quantitative outputs, hybrid approaches (frozen encoder + learned task head) are more sample-efficient. This connects to broader findings in mechanistic interpretability about the gap between what models learn and what they express, and opens a research direction: characterizing which task families suffer from pathway-training deficits and designing encoders or decoders specifically to close them. The equivalence of accuracy across architectures also relates to lottery ticket and mode connectivity literature, hinting that neural networks have high-dimensional symmetries in solution space.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::