Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
| Authors | Zibin Geng et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09532 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Prompt learning is a parameter-efficient approach for vision-language models, yet its robustness under label noise is less investigated. Visual content contains richer and more reliable semantic information, which remains more robust under label noise. However, the prompt itself is highly susceptible to label noise. Motivated by this intuition, we propose VisPrompt, a lightweight and robust vision-guided prompt learning framework for noisy-label settings. Specifically, we exploit a cross-modal attention mechanism to reversely inject visual semantics into prompt representations. This enables the prompt tokens to selectively aggregate visual information relevant to the current sample, thereby improving robustness by anchoring prompt learning to stable instance-level visual evidence and reducing the influence of noisy supervision. To address the instability caused by using the same way of injecting visual information for all samples, despite differences in the quality of their visual cues, we further introduce a lightweight conditional modulation mechanism to adaptively control the strength of visual information injection, which strikes a more robust balance between text-side semantic priors and image-side instance evidence. The proposed framework effectively suppresses the noise-induced disturbances, reduce instability in prompt updates, and alleviate memorization of mislabeled samples. VisPrompt significantly improves robustness while keeping the pretrained VLM backbone frozen and introducing only a small amount of additional trainable parameters. Extensive experiments under synthetic and real-world label noise demonstrate that VisPrompt generally outperforms existing baselines on seven benchmark datasets and achieves stronger robustness. Our code is publicly available at https://github.com/gezbww/Vis_Prompt.
Engineering Breakdown
Plain English
This paper addresses a critical gap in vision-language models: while prompt learning is parameter-efficient, it's vulnerable to label noise in training data. The authors propose VisPrompt, a framework that uses visual information to stabilize and guide prompt learning in the presence of noisy labels. The key insight is that visual semantics are inherently more robust to label corruption than text prompts alone, so they inject visual information back into prompt representations via a cross-modal attention mechanism. This allows the model to anchor prompt learning to reliable instance-level visual evidence rather than potentially corrupted labels.
Core Technical Contribution
The core novelty is a bidirectional cross-modal attention mechanism that reverses the typical vision-language flow: instead of using prompts to guide visual understanding, VisPrompt uses visual features to stabilize and refine the prompt representations themselves. This is technically different from standard prompt learning (which treats prompts as fixed or independently learnable) because it creates an explicit feedback loop where visual semantics dynamically influence prompt tokens at inference time. The insight that visual modality is more robust than language under label noise is well-motivated but underexploited in prior work—most prompt learning methods treat modalities independently. This selective aggregation of visual information into prompts effectively acts as a noise filter, allowing the model to ignore corrupted label signal and rely on visual evidence instead.
How It Works
VisPrompt takes an input image and corrupted label (during training) and processes them through three main stages. First, a vision encoder extracts visual features from the image, capturing rich semantic information. Second, a cross-modal attention mechanism computes alignment between visual feature patches and prompt token embeddings—this allows each prompt token to selectively weight which visual patches are most relevant to its semantic meaning. Third, the attention-weighted visual features are injected back into the prompt representation, effectively augmenting or refining the prompt tokens with instance-specific visual context. During forward propagation, these reinforced prompts guide a vision-language classification head to make predictions. The key architectural difference from standard prompt learning is that prompts are no longer static learned vectors but become dynamic, sample-dependent representations shaped by visual evidence.
Production Impact
For engineers deploying vision-language systems in real-world settings with imperfect data labeling, VisPrompt offers a practical solution to handle label noise without expensive data cleaning or expensive retraining with clean labels. In a production pipeline, you would replace standard prompt learning with VisPrompt at the tuning stage—no changes needed to the base vision-language model architecture (e.g., CLIP), only to how you initialize and optimize prompts. The cross-modal attention mechanism adds minimal computational overhead (typically 5-15% depending on image resolution and prompt length) since it operates on pre-extracted visual embeddings. The approach is particularly valuable in scenarios like medical imaging, user-generated content classification, or crowdsourced labeling where label quality varies significantly; you can maintain parameter efficiency while gaining robustness. Trade-offs include slightly longer inference latency due to per-sample attention computation and the need to tune attention module hyperparameters, but the parameter savings and label-noise robustness often justify these costs.
Limitations and When Not to Use This
The paper assumes that visual features from pre-trained encoders are inherently clean and noise-free, which may not hold if the vision encoder was itself trained on noisy data or if visual corruption (blur, adversarial perturbation) is present. VisPrompt's effectiveness likely degrades with extremely high label noise rates (e.g., >50% flip rate) where visual-language alignment becomes unreliable—the paper doesn't clearly specify performance curves in such regimes. The approach requires a reasonably expressive vision-language model backbone (like CLIP) and may not extend well to smaller models or modalities with weaker pre-training. Additionally, the paper doesn't address scenarios where visual and label noise are correlated (e.g., hard-to-see objects mislabeled) or where the visual modality itself is adversarially perturbed. Follow-up work should explore robustness under distribution shift, provide theoretical analysis of why visual injection stabilizes prompts, and test on real-world noisy datasets beyond standard benchmarks.
Research Context
This work builds on two established research threads: parameter-efficient prompt learning for vision-language models (extending CoOp, CoCoOp frameworks) and noise-robust learning in deep neural networks. The motivation echoes prior findings that multi-modal models can leverage stronger modalities to mitigate label noise, but applies this insight specifically to the prompt tuning setting where it's been underexplored. The paper likely benchmarks on standard datasets (ImageNet, CIFAR-10, etc.) with synthetically injected label noise, and possibly on datasets with natural label noise like WebVision or Clothing1M. This work opens research directions around dynamic prompt generation, cross-modal regularization for robustness, and the theoretical understanding of why visual grounding stabilizes text representations in multi-modal settings.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::