From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
| Authors | Van-Truong Le |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16270 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints \textit{Incorrect Example} and \textit{Misinterpretation} as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.
Engineering Breakdown
Plain English
This paper addresses the critical barrier to public access to justice in Vietnam by evaluating how well state-of-the-art large language models can simplify complex legal texts. The authors benchmark four leading models—GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1—across three performance dimensions: Accuracy, Readability, and Consistency. Beyond aggregate metrics, they conduct a large-scale error analysis on 60 curated Vietnamese legal articles using expert validation to understand why models succeed or fail at this task. The paper delivers both a performance ranking of these models for legal text simplification and actionable insights into their failure modes and capabilities.
Core Technical Contribution
The core contribution is a dual-aspect evaluation framework that goes beyond standard NLP metrics to assess LLM performance on legal text simplification in a non-English language context. Rather than relying solely on BLEU scores or readability indices, the authors introduce an expert-validated error taxonomy derived from analyzing 60 real Vietnamese legal articles, enabling fine-grained diagnosis of model failures. This combination of quantitative benchmarking (Accuracy, Readability, Consistency) with qualitative error analysis represents a more rigorous evaluation methodology than prior work, which typically uses surface-level metrics or English-only datasets. The framework is specifically designed to measure whether models genuinely improve public access to justice—a socially grounded evaluation criterion rarely incorporated into LLM benchmarks.
How It Works
The evaluation pipeline operates in two phases. First, the authors apply four state-of-the-art LLMs to simplify Vietnamese legal texts and measure three dimensions: Accuracy (preservation of legal meaning), Readability (reduction in syntactic/vocabulary complexity), and Consistency (uniform style and terminology across outputs). Second, they conduct a large-scale error analysis by having domain experts annotate a curated dataset of 60 complex legal articles, creating a structured taxonomy of failure modes (e.g., semantic drift, incomplete simplification, introduced inaccuracies). This expert-validated taxonomy becomes the lens through which model outputs are re-examined, allowing investigators to map which error types correlate with which models and which dimensions. The dual-aspect design ensures that raw benchmark scores are grounded in interpretable, actionable error patterns rather than treated as black-box numbers.
Production Impact
For engineers building legal tech or public justice systems, this work provides a production-ready evaluation methodology that prevents deployment of models that appear good on standard metrics but fail on domain-specific requirements. If you were building a legal document simplification service, you could adopt this framework to validate that your chosen model doesn't sacrifice legal accuracy for readability—a common trade-off in simplification tasks. The explicit error taxonomy enables you to prioritize which failure modes are acceptable (e.g., minor stylistic inconsistencies) versus catastrophic (e.g., semantic drift that changes legal obligations), allowing you to set deployment gates based on real-world consequences. The benchmark also informs model selection: these results show which of the four evaluated models dominates across dimensions, saving you from expensive pilot deployments. The main production cost is the requirement for domain expert annotation to validate the error taxonomy—feasible for specialized domains like law but not for every use case.
Limitations and When Not to Use This
The evaluation is limited to Vietnamese legal texts, so findings may not generalize to other languages with different grammatical structures or legal systems; replication in other languages would require re-validating the error taxonomy. The paper evaluates only four specific model versions as of 2026, and LLMs are released at high velocity—results become stale quickly and don't necessarily predict behavior of future model architectures or fine-tuned variants. The expert annotation process, while rigorous, introduces potential subjectivity into error classification and may not capture all failure modes that would surface in full production use with thousands of articles. Finally, the paper measures model outputs in isolation and doesn't address practical deployment challenges like latency, cost per request, or integration with existing legal workflows, which are critical for real justice systems serving large populations.
Research Context
This work builds on prior research in task-specific LLM evaluation and domain adaptation for NLP, where generic benchmarks (like GLUE or SuperGLUE) often miss specialized requirements. It extends recent work on legal NLP by grounding evaluation in actual accessibility goals—prior legal NLP research focused on classification, entity extraction, or case outcome prediction rather than the democratization use case. The paper contributes to a growing recognition that LLM evaluation requires problem-specific frameworks rather than one-size-fits-all metrics; similar multi-dimensional evaluation has been applied to summarization and machine translation but rarely to legal simplification in low-resource languages. This research opens a methodological direction for evaluating LLMs on high-stakes applications where expert feedback and error taxonomy design are central to determining real-world suitability.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::