HSG: Hyperbolic Scene Graph
| Authors | Liyang Wang et al. |
| Year | 2026 |
| HF Upvotes | 0 |
| arXiv | 2604.17454 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Scene graph representations enable structured visual understanding by modeling objects and their relationships, and have been widely used for multiview and 3D scene reasoning. Existing methods such as MSG learn scene graph embeddings in Euclidean space using contrastive learning and attention based association. However, Euclidean geometry does not explicitly capture hierarchical entailment relationships between places and objects, limiting the structural consistency of learned representations. To address this, we propose Hyperbolic Scene Graph (HSG), which learns scene graph embeddings in hyperbolic space where hierarchical relationships are naturally encoded through geometric distance. Our results show that HSG improves hierarchical structure quality while maintaining strong retrieval performance. The largest gains are observed in graph level metrics: HSG achieves a PP IoU of 33.17 and the highest Graph IoU of 33.51, outperforming the best AoMSG variant (25.37) by 8.14, highlighting the effectiveness of hyperbolic representation learning for scene graph modeling. Code: https://github.com/AIGeeksGroup/HSG.
Engineering Breakdown
Plain English
This paper proposes Hyperbolic Scene Graph (HSG), a new method for learning structured representations of 3D scenes and objects by embedding them in hyperbolic space instead of traditional Euclidean space. Scene graphs are graph-structured models that represent objects as nodes and relationships between objects as edges, enabling machines to reason about complex visual scenes. The key insight is that hyperbolic geometry naturally encodes hierarchical relationships — the kind of parent-child or category-subcategory structure that appears in real scenes — through its geometric properties, where distance grows exponentially. The authors show that by training scene graph embeddings in hyperbolic space, they improve the quality of learned hierarchical structures while maintaining strong performance on retrieval tasks compared to prior Euclidean methods like MSG.
Core Technical Contribution
The core novelty is the geometric insight that hyperbolic space is fundamentally better suited for learning hierarchical scene representations than Euclidean space. Rather than using standard contrastive learning in flat Euclidean geometry, HSG exploits the property that hyperbolic geometry expands exponentially, making it ideal for representing nested hierarchies where objects naturally exist at different levels of specificity. The authors demonstrate that by formulating the scene graph embedding problem in hyperbolic space, they can naturally capture entailment relationships — for instance, that a 'chair' is more specific than 'furniture' — without explicitly engineering these constraints. This is a departure from prior methods that learn flat embeddings and then try to impose hierarchy through loss functions or post-hoc alignment.
How It Works
The input is a scene graph structure containing objects (nodes) and their spatial and semantic relationships (edges), along with visual features extracted from multiview or 3D scene data. The method maps each object and relationship into hyperbolic space using the Poincaré model, a standard representation where points live in a bounded disk and distances grow exponentially toward the boundary. During training, rather than minimizing contrastive loss in Euclidean space, the model optimizes embeddings such that hierarchically related objects maintain specific geometric relationships — parent concepts are positioned toward the origin while more specific concepts are pushed toward the boundary, with distance encoding the degree of specificity. The association between objects is learned through attention mechanisms that operate on the hyperbolic embeddings directly, allowing the model to reason about which relationships are consistent with the learned hierarchy. The output is a set of hyperbolic embeddings for all objects and relationships that can be used for downstream tasks like scene retrieval or relationship prediction.
Production Impact
In production systems processing 3D scene understanding or multiview scene reasoning, adopting HSG could improve the structural validity of learned representations, reducing spurious relationships and improving zero-shot generalization to unseen object categories through better hierarchy encoding. Teams building 3D asset management systems, AR/VR scene understanding, or robotics perception pipelines would benefit from more reliable hierarchical reasoning — for example, a robot could more reliably understand that a 'dining chair' is both a 'chair' and 'furniture' without explicit supervision. The main trade-off is computational: hyperbolic operations (especially distance calculations and embeddings in the Poincaré model) are more expensive than Euclidean equivalents, potentially increasing embedding inference latency by 20-40% depending on implementation. Integration complexity is moderate — existing scene graph pipelines would need hyperbolic distance metrics and loss functions swapped in, but the overall architecture remains compatible with standard contrastive learning and attention mechanisms.
Limitations and When Not to Use This
The paper does not address how to handle dynamic scenes where the hierarchy itself changes over time, nor does it provide clear guidance on when hyperbolic embeddings would outperform simpler approaches for flat, non-hierarchical relationships. The method assumes that meaningful hierarchies exist in the data and may not provide benefits for scenes with shallow or ambiguous hierarchical structure — for instance, relationships between objects at the same semantic level (e.g., 'to the left of' spatial relations) may not leverage the hyperbolic geometry effectively. The abstract is truncated and does not provide quantitative results, making it unclear how significant the improvements are in absolute terms or whether they justify the computational overhead in practical settings. Additionally, there is no discussion of how the method handles multiple inheritance (when an object could belong to multiple categories) or conflicting hierarchies, which are common in real-world scene understanding tasks.
Research Context
This work builds on the foundation of scene graphs as a structured representation for visual understanding, extending prior work like Message Passing Scene Graphs (MSG) that use Euclidean embeddings with contrastive learning. The broader research trend it participates in is the adoption of non-Euclidean geometries in deep learning — similar to how hyperbolic embeddings have been successfully applied to knowledge graphs and hierarchical data representation in NLP. The contribution is motivated by observations that many real-world visual domains exhibit natural hierarchies (object taxonomies, spatial containment, semantic abstraction), yet existing methods fail to exploit this structure explicitly. This opens a research direction toward geometry-aware scene representation learning, where different geometric spaces (hyperbolic, spherical, Euclidean) are matched to different relational properties in the data.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::