ESG-Bench: Benchmarking Long-Context ESG Reports for Hallucination Mitigation
| Authors | Siqi Sun et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.13154 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
As corporate responsibility increasingly incorporates environmental, social, and governance (ESG) criteria, ESG reporting is becoming a legal requirement in many regions and a key channel for documenting sustainability practices and assessing firms' long-term and ethical performance. However, the length and complexity of ESG disclosures make them difficult to interpret and automate the analysis reliably. To support scalable and trustworthy analysis, this paper introduces ESG-Bench, a benchmark dataset for ESG report understanding and hallucination mitigation in large language models (LLMs). ESG-Bench contains human-annotated question-answer (QA) pairs grounded in real-world ESG report contexts, with fine-grained labels indicating whether model outputs are factually supported or hallucinated. Framing ESG report analysis as a QA task with verifiability constraints enables systematic evaluation of LLMs' ability to extract and reason over ESG content and provides a new use case: mitigating hallucinations in socially sensitive, compliance-critical settings. We design task-specific Chain-of-Thought (CoT) prompting strategies and fine-tune multiple state-of-the-art LLMs on ESG-Bench using CoT-annotated rationales. Our experiments show that these CoT-based methods substantially outperform standard prompting and direct fine-tuning in reducing hallucinations, and that the gains transfer to existing QA benchmarks beyond the ESG domain.
Engineering Breakdown
Plain English
ESG-Bench is a new benchmark dataset designed to help large language models accurately analyze lengthy corporate ESG (environmental, social, governance) reports while reducing hallucination—instances where models generate plausible-sounding but factually incorrect information. The paper addresses a real business problem: ESG reports are legally mandated in many regions and are critical for assessing corporate sustainability, but they're extremely long and complex, making manual analysis expensive and automated analysis unreliable. The authors created a dataset of human-annotated question-answer pairs grounded in real ESG reports, with fine-grained labels that explicitly mark whether model outputs are factually supported or hallucinated. This enables researchers to benchmark and develop LLMs that can trustworthily extract insights from these documents at scale.
Core Technical Contribution
The core novelty is a purpose-built benchmark dataset specifically targeting the hallucination problem in long-context document understanding, rather than generic QA benchmarks. Unlike existing benchmarks that may assume shorter documents or simpler domains, ESG-Bench reflects real-world constraints: documents are genuinely long and complex, questions span different levels of reasoning (factual retrieval, inference, cross-document synthesis), and annotations explicitly label hallucination instances rather than just marking right/wrong answers. This fine-grained labeling enables detailed error analysis and targeted hallucination mitigation techniques. The contribution shifts from asking "can models answer this question" to the more nuanced question "does the model answer truthfully based on what's actually in the document, or does it fabricate information that sounds plausible but isn't grounded in the text."
How It Works
The ESG-Bench pipeline begins with collecting real-world ESG reports from public corporations, which serve as the source documents that anchor all ground truth. Human annotators then craft question-answer pairs directly from these reports, ensuring each answer is verifiable against the actual text—this prevents the garbage-in-garbage-out problem where synthetic data might contain incorrect answers that models then memorize. Each QA pair is labeled with fine-grained metadata indicating whether the correct answer is explicitly stated in the report, requires inference across multiple sentences, or involves cross-document reasoning. When an LLM generates a response, it's evaluated not just for accuracy but specifically for whether the model's output is factually grounded in the source documents or constitutes a hallucination (plausible but unsupported). This structured annotation enables researchers to measure hallucination rates per question type, per document length, and per model, providing diagnostic insights into where and why hallucinations occur.
Production Impact
For engineers deploying LLMs on ESG analysis, this benchmark enables rigorous evaluation before production deployment—you can measure your model's actual hallucination rate on realistic documents before risking regulatory or compliance failures. In practice, you'd use ESG-Bench to validate that your LLM-powered document analysis system correctly extracts facts (e.g., greenhouse gas emissions figures, board diversity statistics) without fabricating supporting evidence or numbers, which is critical since ESG reports feed into investor decisions and regulatory compliance. The dataset allows you to identify model failure patterns: for instance, you might discover that your model hallucinates when asked about implicit claims or when the true answer requires synthesizing information across 5+ pages, helping you decide whether to use retrieval-augmented generation (RAG), fine-tuning, or chain-of-thought prompting. Integration typically involves: (1) running your current models against the benchmark to establish baseline hallucination rates, (2) identifying which question types trigger hallucinations, (3) iterating on mitigation strategies (better prompts, RAG, model selection), and (4) re-evaluating. The trade-off is that ESG-Bench requires human annotation effort to expand and maintain as new report formats emerge.
Limitations and When Not to Use This
ESG-Bench is limited to the ESG domain and the specific structure of corporate sustainability reports; findings about hallucination patterns may not transfer directly to other long-context domains like medical literature, legal contracts, or financial statements where different types of claims and reasoning patterns dominate. The benchmark is snapshot-in-time: as LLM capabilities and ESG reporting standards evolve, the benchmark may become easier or less representative, requiring periodic re-annotation and expansion. The paper assumes that human annotators can reliably identify hallucinations, but in practice some ESG claims are ambiguous or subject to interpretation, and the benchmark likely cannot capture all such edge cases. Additionally, the benchmark addresses hallucination detection and evaluation but does not prescribe a full solution to prevent hallucinations during generation—it's a measurement tool, not a mitigation guarantee. Finally, the benchmark likely focuses on English-language reports from developed markets; global ESG reporting in other languages and emerging markets is underrepresented, limiting generalization.
Research Context
This work builds on a growing body of research around hallucination in LLMs (prior work like FactKG, TruthfulQA, and FEVER showed that models generate false statements confidently) and extends it into a new, high-stakes domain where accuracy directly impacts corporate governance and investor decision-making. ESG-Bench complements existing long-context benchmarks like LongBench and LLaMA-Index by providing domain-specific, hallucination-annotated data rather than generic retrieval or summarization tasks. The benchmark contributes to the broader shift toward domain-specific evaluation: rather than relying on general-purpose QA datasets, researchers and practitioners now recognize that benchmarks for specialized domains (legal, medical, financial) must reflect real-world constraints and failure modes. This opens up follow-up directions in hallucination mitigation for long documents, improved retrieval-augmented generation specifically for ESG analysis, and domain-specific fine-tuning strategies that trade off model size and latency against hallucination rates in compliance-critical applications.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::