Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs
| Authors | Siyuan Huang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00814 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
While autoregressive Large Vision-Language Models (LVLMs) demonstrate remarkable proficiency in multimodal tasks, they face a "Visual Signal Dilution" phenomenon, where the accumulation of textual history expands the attention partition function, causing visual attention to decay inversely with generated sequence length. To counteract this, we propose Persistent Visual Memory (PVM), a lightweight learnable module designed to ensure sustained, on-demand visual perception. Integrated as a parallel branch alongside the Feed-Forward Network (FFN) in LVLMs, PVM establishes a distance-agnostic retrieval pathway that directly provides visual embeddings for precise visual perception, thereby structurally mitigating the signal suppression inherent to deep generation. Extensive experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead, delivering consistent average accuracy gains across both 4B and 8B scales, particularly in complex reasoning tasks that demand persistent visual perception. Furthermore, in-depth analysis reveals that PVM can resist length-induced signal decay and accelerate internal prediction convergence.
Engineering Breakdown
Plain English
This paper identifies and solves a critical problem in large vision-language models (LVLMs) called Visual Signal Dilution, where the model's ability to perceive visual information degrades as it generates longer sequences of text. The authors propose Persistent Visual Memory (PVM), a lightweight learnable module that runs parallel to the standard feed-forward layers in LVLMs to maintain consistent access to visual embeddings regardless of how long the generated sequence becomes. PVM works by creating a distance-agnostic retrieval pathway that decouples visual perception from the growing attention bottleneck caused by accumulated text tokens. The approach was validated on Qwen3-based models, demonstrating that visual attention can be sustained throughout deep generation without the typical decay pattern.
Core Technical Contribution
The core novelty is the identification and quantification of Visual Signal Dilution as a measurable architectural problem in autoregressive LVLMs—where visual token attention weight decreases inversely with sequence length as the attention partition function expands with textual history. Rather than modifying the main attention mechanism, PVM introduces a parallel auxiliary pathway that bypasses the crowded attention budget entirely, allowing direct retrieval of visual embeddings on-demand. This is structurally different from prior work that attempts to fix visual grounding through better prompting, longer context windows, or modified attention masking patterns. The key insight is that visual information should be retrieved independently via a dedicated learnable module rather than competing in a shared attention space with text tokens.
How It Works
PVM operates as a parallel branch integrated alongside the Feed-Forward Network (FFN) in each transformer block of the LVLM. During forward pass, while the standard attention mechanism processes both visual and textual tokens (with visual weight diminishing over sequence depth), PVM maintains its own lightweight retrieval module that can access the original visual embeddings without attention competition. The module learns a distance-agnostic retrieval function—meaning it retrieves relevant visual information based on semantic similarity rather than position in the sequence, making it robust to long generation. These retrieved visual embeddings are projected and added to the hidden states in parallel with the FFN output, ensuring the model always has fresh visual grounding available. The learnable parameters in PVM (projection weights, retrieval attention heads) are trained end-to-end alongside the main model, allowing the module to learn when and what visual information to retrieve for each generation step.
Production Impact
For engineers deploying LVLMs in production, this solves a real degradation problem: visual question-answering, image captioning, and visual reasoning tasks become increasingly inaccurate as the model generates longer outputs. Integrating PVM into an existing LVLM requires adding a parallel module with minimal compute overhead—likely 5-15% additional parameters depending on implementation, since it's designed as lightweight. The main production benefit is improved output quality on long-form visual reasoning without retraining the base model from scratch; you can likely add PVM to frozen pretrained weights via LoRA-style adaptation. Latency impact should be minimal since retrieval is parallel to FFN computation, and you avoid the expensive option of increasing context window size. The trade-off is that you introduce an additional module to manage and version, and PVM performance depends on quality of the underlying visual embeddings—if your image encoder produces poor representations, PVM has nothing good to retrieve.
Limitations and When Not to Use This
The paper does not address what happens in truly multimodal scenarios with multiple images, videos, or interleaved visual-textual content; PVM's retrieval mechanism assumes a single primary visual embedding bank. The approach is evaluated only on Qwen3-based models, so generalization to other LVLM architectures (e.g., Claude, GPT-4V, LLaVA) remains unexplored and may require architecture-specific tuning. The paper assumes that visual signal decay is the primary problem causing long-sequence visual reasoning failures, but this may not hold in cases where the issue is insufficient visual reasoning capability in the language model itself rather than attention budget constraints. Additionally, there's no analysis of how PVM interacts with popular inference optimizations like KV-cache quantization or speculative decoding, which could affect retrieval efficiency in production deployments. The evaluation likely lacks edge cases like adversarial inputs, out-of-distribution images, or scenarios with ambiguous visual content where independent retrieval could amplify hallucination.
Research Context
This work builds directly on the observation that transformer attention mechanisms suffer from competition between modalities in multimodal settings, related to prior work on cross-attention bottlenecks in vision-language fusion. It contributes to the broader research direction of making LVLMs more robust at long-context visual reasoning, complementing work on extended context windows (like Ring Attention or Sparse Attention patterns) by offering a modality-specific solution. The paper positions itself as a structural fix to a fundamental architectural limitation of decoder-only LVLMs, contrasting with incremental improvements from better prompting or training objectives. This opens future research into other modality-specific persistent memory modules for audio, video, or structured data in multimodal LLMs, suggesting a new design pattern for handling heterogeneous information in long-sequence generation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::