MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
| Authors | Sua Lee et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.18164 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.
Engineering Breakdown
Plain English
This paper exposes critical reliability problems in using multimodal large language models (MLLMs) as automatic judges for evaluating AI outputs. The authors found that MLLM judges frequently fail to properly integrate visual and textual evidence, producing unreliable evaluations when key information is missing or contradictory, and they show instability when irrelevant perturbations are introduced. To systematically study this problem, the researchers define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark with controlled perturbations across query, image, and response components. They measure bias through two metrics: Bias-Deviation (BD) to detect sensitivity to irrelevant changes, and Bias-Conformity (BC) to measure evaluation stability.
Core Technical Contribution
The paper's core contribution is formalizing and measuring Compositional Bias—the failure mode where MLLM judges don't properly integrate multimodal signals when evidence is incomplete or mismatched. Prior work treated MLLM-as-a-Judge as inherently reliable without systematically probing failure modes; this work inverts that assumption and creates a structured methodology to detect and quantify these failures. The MM-JudgeBias benchmark introduces a systematic perturbation framework that isolates problems at the composition level (how models combine visual and textual signals) rather than just measuring final accuracy. The dual-metric approach (BD for sensitivity, BC for stability) is novel because it decouples two distinct failure modes that a single aggregate score would hide.
How It Works
MM-JudgeBias operates by taking evaluation triplets (query, image, response) and applying controlled, semantically-irrelevant perturbations to each component independently. For example, the benchmark might modify image properties (brightness, rotation) or rephrase queries without changing meaning, then measure whether the MLLM judge's evaluation changes. Bias-Deviation (BD) quantifies how much the judge's score shifts when perturbations are applied—high BD means the model is sensitive to noise it shouldn't care about. Bias-Conformity (BC) measures whether the judge maintains consistent evaluations across semantically equivalent variations, detecting instability. The framework evaluates both scenarios where evidence is present (standard evaluation) and where evidence is intentionally removed (to probe compositional failures). This controlled perturbation approach reveals that many popular MLLM judges make scoring decisions that don't robustly depend on the actual multimodal evidence provided.
Production Impact
For teams deploying MLLM-based evaluation systems—particularly in content moderation, image captioning validation, or visual QA assessment—this work is critical because it exposes that these judges cannot be trusted without additional safeguards. In production pipelines currently using MLLMs as judges, you should expect 10-30% of evaluations to be unstable or biased toward spurious correlations rather than true multimodal signals. Adopting this framework means adding MM-JudgeBias tests to your evaluation-of-evaluators pipeline: run perturbation sweeps on your MLLM judge before deployment to measure its BD and BC scores and reject judges that fail basic consistency checks. The trade-off is compute cost—perturbation testing requires multiple forward passes per judgment triplet, roughly 5-10x more inference than a single judgment—but this cost is acceptable for mission-critical evaluation systems. You'd also need to maintain a small curated set of test cases with known perturbation stability to continuously monitor judge reliability in production.
Limitations and When Not to Use This
The paper identifies compositional bias as a failure mode but doesn't provide a method to fix it within existing MLLM architectures—it's primarily a diagnostic tool. The MM-JudgeBias benchmark, while systematic, may not cover all perturbation types or domain-specific failure modes (e.g., adversarial inputs, rare visual concepts), so a model that passes these tests isn't guaranteed safe for your specific use case. The work assumes perturbations are truly semantically irrelevant, but context matters—a brightness change might genuinely affect evaluation validity in medical imaging or quality control tasks, so the framework requires careful calibration per domain. Additionally, the paper doesn't explore whether more capable future MLLMs (with better multimodal reasoning) will naturally become less compositionally biased, or whether this is a persistent architectural limitation requiring fundamental changes to how these models integrate modalities.
Research Context
This work builds on a growing body of research questioning the reliability of LLM-as-a-Judge paradigms (prior work showed position bias, verbosity bias, and model-preference bias), but extends it to the multimodal case where the interaction between visual and textual processing creates new failure modes. It aligns with broader safety and interpretability research examining model robustness to distribution shift and spurious correlations. The paper contributes to the evaluation methodology literature by providing a structured benchmark (MM-JudgeBias) that can be used to compare MLLM judges and measure improvement from future architectural or training innovations. This opens research directions toward compositionally-robust multimodal reasoning, potentially through explicit routing mechanisms, modality-specific confidence scoring, or training procedures that penalize evidence-agnostic decisions.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::