Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning
| Authors | Hui Zhong et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20148 |
| Download | |
| Categories | cs.CV |
Abstract
Automated building facade inspection is a critical component of urban resilience and smart city maintenance. Traditionally, this field has relied on specialized discriminative models (e.g., YOLO, Mask R-CNN) that excel at pixel-level localization but are constrained to passive perception and worse generization without the visual understandng to interpret structural topology. Large Multimodal Models (LMMs) promise a paradigm shift toward active reasoning, yet their application in such high-stakes engineering domains lacks rigorous evaluation standards. To bridge this gap, we introduce a human-in-the-loop semi-automated annotation framework, leveraging expert-proposal verification to unify 12 fragmented datasets into a standardized, hierarchical ontology. Building on this foundation, we present \textit{DefectBench}, the first multi-dimensional benchmark designed to interrogate LMMs beyond basic semantic recognition. \textit{DefectBench} evaluates 18 state-of-the-art (SOTA) LMMs across three escalating cognitive dimensions: Semantic Perception, Spatial Localization, and Generative Geometry Segmentation. Extensive experiments reveal that while current LMMs demonstrate exceptional topological awareness and semantic understanding (effectively diagnosing "what" and "how"), they exhibit significant deficiencies in metric localization precision ("where"). Crucially, however, we validate the viability of zero-shot generative segmentation, showing that general-purpose foundation models can rival specialized supervised networks without domain-specific training. This work provides both a rigorous benchmarking standard and a high-quality open-source database, establishing a new baseline for the advancement of autonomous AI agents in civil engineering.
Engineering Breakdown
Plain English
This paper addresses automated building facade inspection by proposing a human-in-the-loop framework that unifies 12 fragmented datasets into a standardized hierarchical ontology, moving beyond traditional discriminative models like YOLO and Mask R-CNN. The authors leverage Large Multimodal Models (LMMs) to enable active reasoning and structural topology understanding rather than just passive pixel-level detection. The core innovation is combining expert-proposal verification in a semi-automated annotation process with LMMs to achieve better generalization and interpretability for high-stakes engineering inspection tasks. While the abstract is incomplete, the approach promises to bring the reasoning capabilities of LMMs to a domain where current deep learning models lack the visual understanding needed for robust structural analysis.
Core Technical Contribution
The primary contribution is a human-in-the-loop semi-automated annotation framework that bridges the gap between fragmented domain-specific datasets and unified structured knowledge representation. Unlike prior work that applies general-purpose discriminative models or LMMs without rigorous evaluation in engineering domains, this paper introduces a systematic methodology: expert-proposal verification creates a quality-controlled feedback loop where human experts validate and refine model suggestions, which then trains better representations. The hierarchical ontology unifying 12 previously siloed datasets establishes a standardized taxonomy for building facade inspection that enables consistent labeling across sources. The key novelty is recognizing that high-stakes engineering applications need both the reasoning capacity of LMMs and the precision of human expert knowledge, fused through an explicit verification protocol rather than applied independently.
How It Works
The system operates in iterative cycles: (1) an LMM generates initial predictions and interpretations of building facade images, including structural component localization and reasoning about damage patterns or degradation; (2) domain experts review these proposals, correcting errors and enriching annotations with structural domain knowledge; (3) this corrected data feeds back into training, improving the LMM's understanding of building topology and failure modes. The framework ingests images from 12 disparate datasets—which likely have different labeling schemes, image qualities, and damage categories—and normalizes them into a unified hierarchical ontology (e.g., building→facade→wall→crack, with severity and location attributes). The LMM component leverages vision-language pre-training to generate not just bounding boxes or segmentation masks, but reasoned descriptions of structural issues, which experts then verify for correctness and completeness. The output is a cleaner, larger, standardized dataset that can train more reliable inspection systems while maintaining interpretability through the hierarchical structure.
Production Impact
In production facade inspection pipelines, this approach replaces manual inspection or brittle single-model solutions with a system that combines human expertise efficiency and AI scalability. Instead of deploying a YOLO model trained on one dataset and encountering generalization failures on unfamiliar building types or degradation patterns, engineers can use an LMM-based system that reasons about structural relationships and degrades gracefully when encountering out-of-distribution cases. The human-in-the-loop annotation framework reduces the cost of creating inspection datasets: rather than hiring experts to label 10,000 images from scratch, you bootstrap with LMM predictions and experts correct only the errors, potentially reducing annotation time by 40-60% depending on model confidence. Trade-offs include: (1) latency—LMM inference on high-resolution facade images may take 5-15 seconds per image versus <100ms for YOLO, requiring careful deployment design; (2) infrastructure—LMMs are compute-intensive, necessitating GPU resources or API calls that increase operational cost; (3) dependency on expert availability—the system requires periodic expert review cycles to maintain quality, so you cannot fully automate the pipeline.
Limitations and When Not to Use This
The paper does not address real-time inspection scenarios where latency is critical (e.g., drone-based live inspection); LMM inference speed makes this impractical without significant optimization work. The hierarchical ontology approach assumes building damage patterns are consistent enough to be codified in a fixed taxonomy, which may not hold for novel architectural styles, climate-specific degradation, or emerging damage types (e.g., from new materials or environmental stressors). The paper lacks quantitative evaluation benchmarks—there is no reported accuracy, precision/recall, or comparison against YOLO/Mask R-CNN baselines, making it impossible to assess whether the added complexity of LMMs and expert verification actually improves over discriminative models in practice. Additionally, the approach assumes access to domain experts for the verification loop, which may not be feasible for smaller municipalities or in resource-constrained regions, limiting deployment scope.
Research Context
This work sits at the intersection of vision-language models and domain-specific engineering AI, responding to the recent surge of LMM capabilities (GPT-4V, LLaVA, Qwen-VL) being adapted to specialized tasks. It builds on the long tradition of discriminative object detection in computer vision (YOLO, Mask R-CNN, Faster R-CNN) while critiquing their limitations for structural reasoning tasks requiring semantic understanding rather than just localization. The paper advances the emerging subfield of human-in-the-loop machine learning by formalizing the expert-proposal verification protocol as a practical framework for high-stakes applications, echoing work in medical imaging and autonomous driving safety validation. By unifying 12 fragmented datasets into a standardized ontology, it contributes to the data curation and benchmarking infrastructure that the community needs—similar to how COCO and Pascal VOC enabled progress in generic object detection, a standardized building facade dataset could accelerate smart city maintenance research.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::