Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding
| Authors | Eun Woo Im et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2604.13313 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision-Language Models demonstrate remarkable capabilities but often struggle with compositional reasoning, exhibiting vulnerabilities regarding word order and attribute binding. This limitation arises from a scarcity of informative samples needed to differentiate subtle semantic variations during contrastive pretraining. Although hard negative mining offers a promising remedy, existing methods lack explicit mechanisms to dictate which linguistic elements undergo modification. Instead of engineering generative architectures, this study establishes lexical concreteness as a fundamental determinant of negative sample efficacy. Modifying highly concrete terms generates more pronounced structural and visual discrepancies, providing a substantially stronger learning signal. Leveraging this principle, ConcretePlant is proposed to systematically isolate and manipulate perceptually grounded concepts. Analyses of the InfoNCE further reveals a severe gradient imbalance, where easily distinguishable pairs disproportionately overwhelm the optimization process and restrict the bandwidth available for nuanced learning. To resolve this degradation, the Cement loss is formulated utilizing a margin-based approach. By correlating psycholinguistic scores with sample difficulty, this objective dynamically calibrates the penalization applied to individual training pairs. Comprehensive evaluations substantiate these theoretical claims. The integrated framework, designated as Slipform, achieves state-of-the-art accuracy across diverse compositional evaluation benchmarks, general cross-modal retrieval, single and multi label linear probing.
Engineering Breakdown
Plain English
This paper addresses a fundamental weakness in vision-language models: they struggle with compositional reasoning tasks that require understanding word order and binding attributes to objects correctly. The authors discovered that when generating hard negative samples during contrastive pretraining, the linguistic properties of the words being modified dramatically affect learning quality. Specifically, modifying highly concrete terms (like 'dog' or 'red') produces stronger visual and structural discrepancies than modifying abstract terms, yielding significantly better training signals. By systematically leveraging lexical concreteness as a guiding principle for negative mining rather than treating word modifications randomly, the paper demonstrates a principled way to improve compositional reasoning without redesigning the model architecture itself.
Core Technical Contribution
The core contribution is establishing lexical concreteness as an explicit, measurable criterion for selecting which linguistic elements to modify when mining hard negatives during vision-language contrastive pretraining. Prior hard negative mining methods either lacked strategic guidance about which words to perturb or relied on hand-engineered generative architectures; this work identifies that concrete nouns and adjectives create larger semantic and visual gaps than abstract terms, making them more informative training examples. The insight is simple but powerful: a single linguistic property (concreteness) can systematically predict negative sample quality and learning signal strength. This enables engineers to improve model robustness without architectural changes, purely through smarter training data curation.
How It Works
The method operates within a standard vision-language contrastive learning pipeline. Given a paired image-text sample during pretraining, the system generates hard negative text samples by identifying concrete vs. abstract terms in the caption using lexical concreteness metrics (likely derived from psychological/linguistic datasets). When constructing negative examples, the algorithm preferentially modifies highly concrete terms—swapping them with semantically similar but visually distinct alternatives, or altering their attributes (color, count, spatial relation). This creates negative captions that remain plausible and require fine-grained visual understanding to distinguish from positives, generating stronger gradient signals during contrastive loss computation. The image encoder and text encoder simultaneously learn to produce embeddings where correct pairings cluster closely while incorrect pairings (with concrete-term modifications) are pushed far apart, forcing the model to learn finer compositional distinctions.
Production Impact
For teams deploying vision-language models in production, this approach directly improves robustness on compositional reasoning benchmarks without requiring model retraining from scratch or architectural refactoring. The practical benefit surfaces in applications like visual search, image captioning verification, and multimodal retrieval where subtle attribute changes matter—e.g., distinguishing 'red car on left' from 'blue car on right' correctly. Implementation requires scoring terms in your training captions by concreteness (a one-time preprocessing step) and modifying your negative sampling logic during pretraining; no inference-time changes are needed, keeping deployment costs flat. The trade-off is primarily in pretraining compute: curating concreteness-aware hard negatives adds overhead during data preparation and sampling, but this one-time cost pays dividends across all downstream tasks relying on that pretrained model.
Limitations and When Not to Use This
The paper assumes access to reliable lexical concreteness scoring, which may not exist or transfer across languages and specialized domains; a medical imaging model, for example, might need task-specific concreteness definitions. The approach is most effective during large-scale contrastive pretraining where hard negative mining is feasible; smaller or fine-tuned models may see minimal gains if data is already saturated or negatives are naturally hard. The paper does not address whether concreteness-based mining helps with other failure modes (like spatial reasoning, counting, or temporal understanding) equally well, leaving open questions about the generality of the principle. Finally, the work depends on having sufficient diversity in concrete terms within your training captions; vocabularies with limited concrete vocabulary may see diminishing returns.
Research Context
This work builds on the established finding that vision-language models (like CLIP) have systematic weaknesses in compositional generalization, a problem studied extensively in papers on attribute binding and word order sensitivity. It advances beyond prior hard negative mining strategies by grounding the selection criterion in linguistic theory rather than heuristics, connecting vision-language robustness to well-studied properties from cognitive science and NLP. The paper implicitly benchmarks against standard compositional reasoning datasets (likely including tasks like distinguishing object-attribute pairs or spatial relations) where recent VLMs have shown measurable failure rates. This opens a research direction toward linguistically-informed training data construction, where semantic and cognitive properties of language guide vision-language learning rather than random perturbations or learned sampling strategies.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::