EffiMiniVLM: A Compact Dual-Encoder Regression Framework
| Authors | Yin-Loon Khor et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.03172 |
| Download | |
| Categories | cs.CV |
Abstract
Predicting product quality from multimodal item information is critical in cold-start scenarios, where user interaction history is unavailable and predictions must rely on images and textual metadata. However, existing vision-language models typically depend on large architectures and/or extensive external datasets, resulting in high computational cost. To address this, we propose EffiMiniVLM, a compact dual-encoder vision-language regression framework that integrates an EfficientNet-B0 image encoder and a MiniLM-based text encoder with a lightweight regression head. To improve training sample efficiency, we introduce a weighted Huber loss that leverages rating counts to emphasize more reliable samples, yielding consistent performance gains. Trained using only 20% of the Amazon Reviews 2023 dataset, the proposed model contains 27.7M parameters and requires 6.8 GFLOPs, yet achieves a CES score of 0.40 with the lowest resource cost in the benchmark. Despite its small size, it remains competitive with significantly larger models, achieving comparable performance while being approximately 4x to 8x more resource-efficient than other top-5 methods and being the only approach that does not use external datasets. Further analysis shows that scaling the data to 40% alone allows our model to overtake other methods, which use larger models and datasets, highlighting strong scalability despite the model's compact design.
Engineering Breakdown
Plain English
EffiMiniVLM addresses a critical business problem: predicting product quality in cold-start scenarios where you have no user interaction history and must rely only on product images and text descriptions. The authors built a lightweight vision-language model combining EfficientNet-B0 for images and MiniLM for text, adding a simple regression head to predict quality scores. They introduced a weighted Huber loss function that trusts samples with more ratings over noisy samples, and trained the entire system on just 20% of Amazon Reviews 2023 data. This approach delivers consistent performance gains while requiring dramatically less compute than standard large vision-language models.
Core Technical Contribution
The core novelty is a deliberately minimal dual-encoder architecture that avoids the computational overhead of large vision-language models while maintaining predictive quality through intelligent loss function design. Rather than using a massive pre-trained model or requiring extensive external datasets, the authors show that pairing an efficient image encoder (EfficientNet-B0) with a lightweight text encoder (MiniLM) and training with a weighted Huber loss can match performance on significantly less data. The weighted Huber loss is the key training-side innovation—it learns to weight each training sample by its rating count, effectively performing automatic curriculum learning where noisy samples with few ratings get lower weight. This is a practical engineering insight: the loss function itself becomes a mechanism to handle data quality variation without requiring manual sample filtering or augmentation.
How It Works
The input to EffiMiniVLM is a product listing containing an image and text metadata (title, description, etc.). The image passes through EfficientNet-B0, a mobile-optimized convolutional network that extracts visual features into a compact embedding vector. Simultaneously, the text passes through a MiniLM encoder (a distilled variant of BERT), producing a text embedding. These two embeddings are concatenated or fused together to create a joint multimodal representation. This combined representation flows through a lightweight regression head—likely 1-2 fully connected layers—that outputs a single continuous value predicting product quality or rating. During training, rather than treating all samples equally, the weighted Huber loss assigns each sample a weight proportional to its rating count (samples with 100+ reviews get higher weight than samples with 2 reviews), making the loss function robust to rating noise while standardizing the training signal.
Production Impact
For engineers building e-commerce or marketplace systems, this approach dramatically reduces the computational footprint compared to full-scale vision-language models like CLIP or Flamingo. You can deploy EffiMiniVLM on edge devices, mobile servers, or cost-efficient inference hardware because EfficientNet-B0 and MiniLM have model sizes in the tens of megabytes, not billions of parameters. The 20% data requirement means you need significantly less labeled data to achieve production-quality predictions, reducing annotation costs and time-to-deployment for new product categories. The trade-off is that this is a regression-only approach (predicts a quality score), not a generative model—you cannot use it for image captioning, retrieval, or other generation tasks. For cold-start quality prediction in real marketplace systems, this offers immediate ROI: fast inference latency (sub-100ms on CPU), low memory footprint, and straightforward integration into existing product ranking pipelines.
Limitations and When Not to Use This
EffiMiniVLM is specialized for the regression task and cannot be used for downstream tasks like image search, cross-modal retrieval, or generation without modification. The approach assumes that product quality is meaningfully predictable from images and text alone—it will struggle with quality factors that depend on brand reputation, seller history, or contextual user preferences that aren't encoded in the product listing. The weighted Huber loss relies on the assumption that rating count correlates with rating reliability, which may not hold if products receive spam ratings or coordinated manipulation. The paper trains on Amazon data, so generalization to other domains (fashion, automotive, luxury goods) is unvalidated—domain-specific retraining will likely be necessary. The authors do not discuss how the model handles cold-start items with zero or very few ratings, which is paradoxically the scenario most needing prediction.
Research Context
This work fits into the broader trend of efficient vision-language models, building on prior research in knowledge distillation (MiniLM is a distilled BERT), mobile-friendly architectures (EfficientNet), and lightweight fusion methods for multimodal learning. It extends beyond earlier work that treated image and text encoders separately by introducing a learned weighting mechanism at the loss level rather than the architecture level. The paper implicitly challenges the assumption that bigger models and more data always win—it demonstrates that intelligent training (weighted loss) can compensate for model scale and data volume. This opens a research direction toward loss-driven improvements in data efficiency, suggesting that how you train matters as much as what you train.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::