FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios
| Authors | Xiangru Jian et al. |
| Year | 2026 |
| HF Upvotes | 81 |
| arXiv | 2604.07413 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.
Engineering Breakdown
Plain English
FORGE addresses a critical gap in evaluating Multimodal Large Language Models (MLLMs) for manufacturing tasks—a domain where current benchmarks fail to capture real-world complexity. The authors built a high-quality multimodal dataset combining 2D images and 3D point clouds with fine-grained domain annotations (like exact component model numbers), then benchmarked 18 state-of-the-art MLLMs across three manufacturing tasks: workpiece verification, structural surface inspection, and assembly verification. The evaluation revealed significant performance gaps between existing MLLMs and the requirements of manufacturing environments, suggesting that models trained on general vision-language data struggle with the precision and specificity needed for industrial applications. This work establishes both the dataset and baseline results that expose where current models fail in manufacturing contexts.
Core Technical Contribution
The core contribution is FORGE, a purpose-built evaluation framework specifically designed for manufacturing scenarios that combines multimodal data (2D + 3D) with fine-grained domain semantics rather than generic annotations. Unlike existing vision-language benchmarks (COCO, Conceptual Captions) that focus on everyday objects and scenes, FORGE grounds evaluation in industrial tasks that require precise component identification and defect detection. The dataset and evaluation methodology directly address the data scarcity problem in manufacturing ML—previous work either used synthetic data or small proprietary datasets, but FORGE establishes a reusable, annotated benchmark with real-world manufacturing imagery. By evaluating 18 different MLLMs, the paper provides empirical evidence of the specific capability gaps that need to be addressed for manufacturing deployment.
How It Works
The FORGE evaluation pipeline operates in three stages: (1) data collection and annotation, where real manufacturing environments are captured as both 2D RGB images and 3D point cloud data, with domain-specific labels like component model numbers and defect categories; (2) multimodal encoding, where MLLMs process both image and point cloud inputs, requiring models to fuse 2D and 3D spatial information; (3) task-specific evaluation on three concrete manufacturing tasks—workpiece verification (confirming correct part arrival), structural surface inspection (detecting defects or deviations), and assembly verification (validating correct assembly state). For each task, the MLLM must produce outputs that match fine-grained manufacturing semantics: exact model numbers, precise defect locations, assembly correctness verdicts. The evaluation methodology likely compares model predictions against ground-truth annotations across metrics like accuracy, F1 score, and task-specific measures that account for the cost of false negatives in manufacturing (a missed defect is costly; a false alarm is less critical). The key technical challenge is that 3D point cloud processing typically requires specialized encoders (PointNet-style architectures), which most general-purpose MLLMs weren't designed to handle, creating a mismatch between model architecture and input modality.
Production Impact
For teams building manufacturing quality-control systems, FORGE provides two immediate value contributions: a reusable evaluation benchmark that predicts real-world MLLM performance in your application, and empirical evidence showing which models to prioritize or rule out. If you're integrating MLLMs into a production inspection pipeline, this paper's failure analysis tells you that generic vision models will likely under-perform on fine-grained component recognition and defect localization—you'll need either domain-specific fine-tuning, custom 3D encoders, or hybrid approaches combining MLLMs with traditional computer vision. The compute trade-off is significant: processing both 2D images and 3D point clouds increases inference latency and memory requirements compared to image-only models, so you'll need to evaluate whether the accuracy gains justify the 2-4x increase in per-frame processing cost. The dataset itself becomes a strategic asset—teams can use it for internal benchmarking, fine-tuning language models on manufacturing tasks, or developing multimodal fusion architectures. One practical consideration: FORGE's fine-grained annotations (exact model numbers, precise defect types) require domain expertise and careful labeling, which is a bottleneck for scaling this approach to new manufacturing domains or parts.
Limitations and When Not to Use This
The paper's scope is limited to three specific manufacturing tasks, which may not generalize to other industrial scenarios like process monitoring, predictive maintenance, or supply chain logistics that have different data characteristics and semantic requirements. FORGE relies on having high-quality 3D point cloud annotations, which is expensive to generate and maintain—the paper doesn't clarify how much human annotation effort was required or provide guidance on scaling to new product types with different geometries. The evaluation is retrospective (testing on a static dataset) rather than prospective, so it doesn't capture how model performance degrades under real-world distribution shifts like lighting variations, sensor miscalibration, or novel defect types never seen during dataset creation—this is crucial for manufacturing where novel failure modes constantly emerge. Additionally, the paper likely assumes that the 18 evaluated MLLMs can actually process 3D point clouds as native inputs or require conversion/adaptation, but many weren't designed for this modality, so the comparison may be unfair if models must be substantially modified to run on FORGE tasks. The fine-grained domain semantics (exact model numbers) will require frequent dataset updates as manufacturing lines introduce new parts, creating ongoing annotation burden that may not be sustainable for all manufacturers.
Research Context
FORGE builds on the tradition of domain-specific benchmarking (similar to how ImageNet revolutionized computer vision evaluation) but applies it to the emerging intersection of multimodal learning and industrial AI. The work addresses a clear gap between general-purpose MLLM benchmarks (like MMVP, MMBench) which focus on everyday scenes, and the real-world industrial applications that require precise, semantically-grounded understanding of manufactured components. It extends prior manufacturing vision work (which typically used single-modality 2D images or specialized 3D deep learning models) by evaluating unified multimodal models that could eventually replace domain-specific pipelines. This paper likely opens a research direction in fine-grained multimodal evaluation for other high-stakes domains (healthcare imaging, autonomous driving, robotics) where generic benchmarks fail to capture domain-specific accuracy requirements and the cost of failure is high.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::