MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
| Authors | Han Wang et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.13418 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.
Engineering Breakdown
Plain English
This paper introduces MERRIN, a benchmark for evaluating search-augmented AI agents on their ability to handle real-world web search tasks. The key problem is that existing benchmarks don't capture how modern search engines return heterogeneous, multimodal, and often conflicting results (text, images, video, audio) in response to underspecified queries. MERRIN is human-annotated and tests whether agents can identify which modalities are relevant, retrieve multimodal evidence across web sources, and reason through multiple steps over noisy data. This moves beyond prior work by omitting explicit modality cues in queries, incorporating underexplored modalities like video and audio, and forcing agents to work with genuinely messy, real-world web content rather than clean curated datasets.
Core Technical Contribution
The core novelty is a systematically designed benchmark that mirrors real search-engine conditions: multimodal queries without explicit format hints, heterogeneous web results spanning multiple modalities, and deliberately conflicting or noisy evidence that requires multi-hop reasoning to resolve. Unlike prior search-augmented agent benchmarks (such as those focusing on single-modality text retrieval or clean structured data), MERRIN enforces end-to-end evaluation of modality selection, evidence gathering, and reasoning under realistic noise. The contribution is primarily in evaluation methodology and dataset design rather than a new algorithmic technique—the paper creates a standard against which to measure agent robustness and reasoning capability in multimodal web environments. This addresses a documented gap: existing benchmarks either ignore modality diversity, use artificially clean data, or provide explicit task-specific modality signals that don't reflect how humans formulate actual search queries.
How It Works
MERRIN operates as a human-annotated benchmark with a three-stage pipeline. First, evaluators create natural language search queries without specifying which modalities (text, image, video, audio) should be used, mirroring real user behavior. Second, the benchmark ingests multimodal web results—pages with text, embedded media, transcripts, etc.—and deliberately includes noisy, conflicting, or irrelevant content to simulate realistic search engine output. Third, an agent must (a) determine which modalities contain relevant evidence, (b) retrieve and rank that evidence across sources, and (c) synthesize multi-hop reasoning—for example, combining a video transcript with text from multiple pages to answer a complex query. The benchmark includes human-annotated ground truth for relevant modalities and correct reasoning chains, allowing metrics to measure both precision in modality selection and correctness of downstream reasoning. Agents are evaluated on their ability to reject misleading content, prioritize high-signal sources, and produce justified answers despite conflicting evidence.
Production Impact
For production search-augmented systems, MERRIN provides a concrete way to measure agent robustness before deployment to users. Teams building multimodal retrieval systems (e-commerce search, medical information lookup, news aggregation) can use MERRIN to identify failure modes: agents that blindly retrieve all modalities incur compute overhead and latency without benefit; agents that ignore video/audio miss relevant information; agents that can't handle conflicting sources produce unreliable answers. Adopting MERRIN-style evaluation would require instrumenting search pipelines to track which modalities are queried and how evidence is ranked, adding observability but minimal latency overhead. The trade-off is data cost: creating human annotations for multimodal queries is expensive, so teams would likely start with a subset and gradually expand coverage. The concrete impact is reducing post-deployment failures where agents return answers contradicted by other web sources or miss high-confidence information because they didn't retrieve the right modality.
Limitations and When Not to Use This
MERRIN is constrained by its reliance on human annotation, which limits scale and introduces annotation bias—different annotators may disagree on which modalities are truly relevant or how to resolve conflicting evidence. The paper doesn't address how to evaluate agents in near-real-time scenarios where web content changes constantly or where modalities (e.g., live video streams) are updated faster than agents can reason over them. The benchmark assumes a static snapshot of web results, which sidesteps the challenge of handling temporal dynamics or stale information. Additionally, the paper evaluates 'diverse search' agents but the abstract cuts off before showing which agent architectures were tested or how different model sizes and retrieval strategies performed—leaving open questions about which design patterns actually excel on MERRIN. The approach also doesn't fully address adversarial cases where noisy content is deliberately designed to mislead (e.g., misinformation in video captions paired with plausible text) or multi-language scenarios common in global search.
Research Context
MERRIN builds on a lineage of search-augmented agent benchmarks (such as HotpotQA for multi-hop reasoning and MultimodalQA) but extends them by combining underspecified queries, real-world noise, and modality diversity. Prior work either focused on single-modality retrieval (text-only) or used artificially clean multimodal datasets where modality relevance was pre-labeled or obvious. The paper also advances the broader trend of evaluating LLM-based agents in realistic environments rather than controlled settings—similar to how benchmarks like WebArena moved from synthetic web environments to real websites. MERRIN opens research directions in modality selection heuristics, uncertainty quantification over conflicting sources, and agent interpretability (i.e., explaining why an agent chose certain modalities). The benchmark is particularly timely given increased focus on retrieval-augmented generation (RAG) and multimodal foundation models; as agents become more capable, evaluation standards must raise to catch subtle failure modes in reasoning over heterogeneous evidence.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::