Modeling Subjective Urban Perception with Human Gaze
| Authors | Lin Che et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00764 |
| Download | |
| Categories | cs.CV, cs.HC |
Abstract
Urban perception describes how people subjectively evaluate urban environments, shaping how cities are experienced and understood. Existing computational approaches primarily model urban perception directly from street view images, but largely ignore the human perceptual process through which such judgments are formed. In this paper, we introduce Place Pulse-Gaze, an urban perception dataset that augments street view images with synchronized eye-tracking recordings and individual perception labels. Based on this dataset, we propose a Gaze-Guided Urban Perception Framework to study how gaze behavior contributes to the modeling of subjective urban perception. The framework systematically investigates three complementary settings: gaze-only modeling, gaze fusion with explicit semantic scene representations, and gaze fusion with implicit richer visual representations. Experiments show that gaze alone already carries useful predictive signals for subjective urban perception, and that integrating gaze with scene representations further improves prediction under both semantic and richer visual representations. Overall, our findings highlight the importance of incorporating human perceptual processes into urban scene understanding and open a direction for gaze-guided multimodal urban computing.
Engineering Breakdown
Plain English
This paper addresses how people subjectively perceive and evaluate urban environments by introducing a new dataset called Place Pulse-Gaze that pairs street view images with eye-tracking recordings and human perception labels. Rather than trying to predict urban perception directly from images like prior work, the authors propose a Gaze-Guided Urban Perception Framework that explicitly models where and how people look when forming judgments about cities. The key insight is that human gaze patterns reveal the perceptual process underlying subjective urban evaluation, enabling the model to learn from these attention patterns rather than working blindly from pixel data. The framework investigates three complementary modeling approaches: using gaze alone, fusing gaze with semantic scene understanding, and presumably end-to-end integration of these signals.
Core Technical Contribution
The paper's core novelty is the systematic integration of human gaze as an explicit signal for modeling subjective perception, rather than treating it as latent or irrelevant. The authors created the first large-scale dataset linking eye-tracking data with urban perception labels and street view images, enabling supervised learning of gaze-perception relationships. The Gaze-Guided Urban Perception Framework is a multi-pathway architecture that treats gaze as a rich supervisory signal—not just a post-hoc explanation of what the model learned, but as an active input that shapes how the model learns to evaluate urban spaces. This represents a fundamental shift from image-only modeling to multimodal perception modeling that respects the human visual attention process.
How It Works
The system takes street view images as input and synchronously captures human eye-tracking data while subjects rate urban perception attributes (e.g., safety, vibrancy, walkability). The framework operates in three complementary modes: (1) Gaze-only modeling, where eye-tracking heatmaps or gaze trajectories directly predict perception scores, likely using CNN or transformer architectures that process gaze saliency maps; (2) Gaze-fusion, where gaze attention is concatenated or cross-attended with explicit semantic scene features (object detections, scene labels) extracted from the image; (3) An integrated setting that learns end-to-end relationships between gaze, semantics, and perception outputs. The gaze data serves as both a training signal (constraining where the model should focus) and a feature representation (encoding which image regions humans find perceptually relevant). Outputs are continuous or categorical perception scores for various urban quality dimensions.
Production Impact
For production urban analytics systems, this work enables more human-aligned perception modeling—critical for city planning tools, real estate valuation, and navigation apps where understanding subjective experience matters. Instead of training black-box models on images alone, you could augment your training pipeline with crowdsourced gaze data, making models interpretable and aligned with actual human attention patterns. The trade-off is substantial: you need eye-tracking hardware or software (e.g., webcam-based gaze estimation) to collect training data, increasing annotation costs and complexity, though inference only requires images. A practical deployment might use gaze-aware training on a smaller, high-quality labeled set, then fine-tune on larger image-only datasets, or deploy gaze prediction as a middle layer to improve urban perception scores without gaze at inference time. Integration with mapping platforms would require connecting eye-tracking data collection pipelines, perception models, and geographic information systems—a non-trivial engineering effort but valuable for applications like urban accessibility assessment or gentrification monitoring.
Limitations and When Not to Use This
The paper's scope is limited by the gaze dataset size and diversity—eye-tracking studies are expensive and typically conducted in labs or with small in-field samples, so the generalization to diverse geographic regions, cultural contexts, and populations remains unclear. The approach assumes that gaze patterns are consistent and meaningful across individuals, but subjective perception is highly variable; two people with identical gaze might make different judgments, and the model may conflate gaze patterns with confounding factors (e.g., age or familiarity). Practical deployment is hampered by the need for gaze data during training; models trained only on gaze supervision may not transfer to image-only inference settings without significant domain adaptation, and the paper's abstract suggests incomplete technical details on how this is handled. Additionally, the work doesn't address whether gaze-aware models produce more accurate predictions overall or merely more interpretable ones—gaze may be correlated with perception without being causally informative, and the framework may not outperform simpler semantic approaches on held-out test sets from new cities or cultures.
Research Context
This work builds on the Place Pulse project, which pioneered computational urban perception modeling from street view imagery, and extends it by incorporating the human perceptual mechanism—a shift toward more cognitively grounded computer vision. It connects to the broader literature on attention mechanisms in deep learning (transformers, spatial attention), visual saliency prediction, and human-in-the-loop machine learning, applying these ideas to the novel domain of subjective urban evaluation. The dataset contribution is significant for the urban computing and HCI communities, providing a rare multimodal resource linking perceptual labels, gaze, and street imagery that will likely enable follow-up work on interpretability, transfer learning, and personalized perception models. The research opens up future directions in cross-cultural gaze studies, the role of gaze in other subjective perception tasks (aesthetics, emotion recognition), and whether gaze-aware pretraining improves downstream urban analytics tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::