MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 69 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Xiyu Ren et al. |
| Year | 2026 |
| HF Upvotes | 69 |
| arXiv | 2605.14906 |
| Download | |
| Code | https://github.com/xrenaf/MEMLENS |
Abstract
Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.
Engineering Breakdown
Plain English
MemLens is a benchmark that tests how well vision-language models handle long, multimodal conversations across multiple sessions. The benchmark includes 789 questions testing five memory abilities (extracting info, reasoning across sessions, temporal understanding, updating knowledge, and knowing when to refuse answers) at context lengths from 32K to 256K tokens, and their experiments confirm that solving these tasks actually requires using the visual evidence—it's not just language understanding.
Key Engineering Insight
There's a meaningful gap between long-context LVLMs (models that just extend context windows) and memory-augmented agents (models with explicit memory modules), but no one had measured which approach actually performs better on multimodal tasks that require real memory. This benchmark reveals which architectural choice works better in production, which is critical for deciding how to build your system.
Why It Matters for Engineers
If you're building a production system that needs to handle long conversations with images—like a document analysis tool, image-based customer service system, or video understanding agent—you need to decide: do you extend context or add memory modules? This benchmark gives you empirical data on which approach scales better and which memory abilities are actually bottlenecks, so you can optimize your architecture before shipping.
Research Context
Previous work studied long-context models and memory-augmented agents separately, but no one systematically compared them on tasks requiring genuine multimodal evidence across sessions. MemLens closes this gap by providing the first standardized benchmark for this comparison, enabling the community to measure progress on a well-defined problem and helping engineers choose between architectural approaches based on empirical evidence rather than speculation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::