FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification
| Authors | Hang Xu et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.04074 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Peer review in machine learning is under growing pressure from rising submission volume and limited reviewer time. Most LLM-based reviewing systems read only the manuscript and generate comments from the paper's own narrative. This makes their outputs sensitive to presentation quality and leaves them weak when the evidence needed for review lies in related work or released code. We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification. Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository under bounded budgets to test central empirical claims. It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive. In a case study on CompGCN, FactReview reproduces results that closely match those reported for link prediction and node classification, yet also shows that the paper's broader performance claim across tasks is not fully sustained: on MUTAG graph classification, the reproduced result is 88.4%, whereas the strongest baseline reported in the paper remains 92.6%. The claim is therefore only partially supported. More broadly, this case suggests that AI is most useful in peer review not as a final decision-maker, but as a tool for gathering evidence and helping reviewers produce more evidence-grounded assessments. The code is public at https://github.com/DEFENSE-SEU/Review-Assistant.
Engineering Breakdown
Plain English
FactReview addresses the peer review bottleneck in machine learning by building a system that goes beyond reading just the manuscript text. Instead of relying solely on the paper's narrative, it extracts claims, retrieves relevant related work to contextualize the contribution, and actually runs released code under resource constraints to verify empirical claims. This multi-stage approach makes review assessments more robust to presentation quality and catches issues that would be invisible to text-only LLM reviewers. The system targets a real operational problem: growing ML submission volumes are overwhelming human reviewers, and most current LLM-based systems lack the grounding needed for thorough technical validation.
Core Technical Contribution
The core innovation is an evidence-grounded review pipeline that combines three distinct capabilities: automated claim extraction from papers, literature retrieval to establish technical positioning, and execution-based verification that actually runs code to test reported results. This differs from prior LLM-based review systems which treat papers as isolated text documents and generate feedback purely from content analysis. FactReview's key insight is that the strongest evidence for validating ML claims exists outside the manuscript—in related work context and in reproducible code execution. The system introduces bounded execution as a practical mechanism to ground reviews in empirical reality rather than narrative plausibility.
How It Works
FactReview operates in three sequential stages. First, it parses the submission manuscript to identify and extract major claims (e.g., 'our method achieves 95% accuracy on benchmark X') and reported empirical results. Second, it uses literature retrieval to fetch nearby published work that contextualizes the paper's technical contribution, helping establish whether claims are truly novel or incremental. Third, when code is released alongside the submission, the system executes the repository under controlled resource budgets (bounded compute, memory, and wall-clock time) to actually run the experiments and verify that reported results are reproducible and claims are substantiated. The outputs from all three stages feed into a final review generation step that produces grounded, evidence-backed comments rather than text-only critiques.
Production Impact
For organizations running peer review pipelines or automated assessment systems, FactReview would fundamentally change how submissions are vetted: instead of one-pass text analysis, you'd have multi-modal evidence including execution traces and literature context. This dramatically reduces false positives (accepting papers with unjustified claims) and makes reviews harder to game through narrative manipulation alone. The production trade-off is significant computational cost—executing arbitrary code under bounded resources requires sandboxing infrastructure, failure handling, and timeout management, adding 5-30 minutes of latency per submission depending on code complexity. Integration requires standardized code release workflows and handling cases where code isn't provided (reverting to the claim-extraction and literature stages). The payoff is reviews that catch reproducibility issues, unjustified comparisons, and overclaimed results before they enter the literature.
Limitations and When Not to Use This
FactReview assumes code is available and well-structured enough to execute in a sandbox, which doesn't hold for many submissions—papers with proprietary datasets, custom hardware dependencies, or missing dependencies will fall back to weaker verification. The bounded execution approach can miss subtle bugs or only sample a small fraction of a paper's claims, so it's a consistency check rather than comprehensive validation. Literature retrieval quality depends on the paper being in a well-indexed domain; for emerging areas with sparse prior work, positioning claims become harder to ground. The system doesn't address other critical review dimensions like clarity, writing quality, or broader impact assessment—it's narrowly focused on factual claim verification. Most importantly, actual code execution can be unreliable or computationally expensive for papers involving large-scale training, reinforcement learning, or interactive systems.
Research Context
This work builds on a decade of research in automated and LLM-assisted peer review, extending recent systems that use language models to generate review feedback. It incorporates insights from the reproducibility crisis in ML—the recognition that many published results cannot be reproduced—and pairs that with practical code execution techniques developed in software engineering verification. FactReview sits at the intersection of claim verification (a growing area in fact-checking systems), scientific document understanding (extracting structured information from research papers), and code-to-text grounding (using code execution to validate natural language claims). The approach opens future research directions in continuous reproducibility checking, automated benchmarking validation, and adversarial robustness of review systems against authors who might game execution-based verification.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::