Terminology Rarity Predicts Catastrophic Failure in LLM Translation of Low-Resource Ancient Languages: Evidence from Ancient Greek
| Authors | James L. Zainaldin et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.24119 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
This study presents the first systematic, reference-free human evaluation of large language model (LLM) machine translation (MT) for Ancient Greek (AG) technical prose. We evaluate translations by three commercial LLMs (Claude, Gemini, ChatGPT) of twenty paragraph-length passages from two works by the Greek physician Galen of Pergamum (ca. 129-216 CE): On Mixtures, which has two published English translations, and On the Composition of Drugs according to Kinds, which has never been fully translated into English. We assess translation quality using both standard automated evaluation metrics (BLEU, chrF++, METEOR, ROUGE-L, BERTScore, COMET, BLEURT) and expert human evaluation via a modified Multidimensional Quality Metrics (MQM) framework applied to all 60 translations by a team of domain specialists. On the previously translated expository text, LLMs achieved high translation quality (mean MQM score 95.2/100), with performance approaching expert level. On the untranslated pharmacological text, aggregate quality was lower (79.9/100) but with high variance driven by two passages presenting extreme terminological density; excluding these, scores converged to within 4 points of the translated text. Terminology rarity, operationalized via corpus frequency in the literary Diorisis Ancient Greek Corpus, emerged as a strong predictor of translation failure (r = -.97 for passage-level quality on the untranslated text). Automated metrics showed moderate correlation with human judgment overall on the text with a wide quality spread (Composition), but no metric discriminated among high-quality translations. We discuss implications for the use of LLMs in Classical scholarship and for the design of automated evaluation pipelines for low-resource ancient languages.
Engineering Breakdown
Plain English
This paper evaluates how well three commercial LLMs (Claude, Gemini, ChatGPT) translate Ancient Greek medical texts, specifically 20 passages from Galen's works totaling 60 translations. The authors use both automated metrics (BLEU, chrF++, METEOR, ROUGE-L, BERTScore, COMET, BLEURT) and expert human evaluation with a modified MQM framework to assess translation quality. The core finding is that terminology rarity—how frequently technical or specialized terms appear in the LLM's training data—predicts catastrophic failure modes where translations become completely unintelligible. This is the first systematic, reference-free evaluation of LLM machine translation for low-resource ancient languages, revealing a fundamental brittleness in how LLMs handle out-of-distribution vocabulary.
Core Technical Contribution
The paper's primary contribution is identifying and quantifying a direct causal relationship between terminology rarity and translation collapse in LLMs for low-resource ancient languages. Rather than assuming LLMs degrade gracefully on unfamiliar terms, the authors demonstrate that below a certain frequency threshold in training data, translations shift from poor-but-functional to completely nonsensical—a discontinuous failure mode. They establish this through systematic human evaluation (not just automated metrics) across multiple commercial models, showing this is not a quirk of one implementation but a fundamental property of how transformer-based LLMs handle rare terminology. This moves beyond prior work that focused on BLEU scores for modern language pairs; the authors pioneer methodology for diagnosing catastrophic failure in ancient language translation.
How It Works
The evaluation pipeline works as follows: (1) Input stage—select 20 paragraph-length passages from Galen's On Mixtures and On the Composition of Drugs by Kinds, totaling ~60 translation instances across three LLMs. (2) Translation generation—feed Ancient Greek source text to Claude, Gemini, and ChatGPT via their APIs, generating target English translations. (3) Automated metrics stage—compute seven standard metrics (BLEU, chrF++, METEOR, ROUGE-L, BERTScore, COMET, BLEURT) against reference translations where available (On Mixtures has two published English translations; the other work has none, requiring reference-free evaluation). (4) Human evaluation stage—apply a modified MQM (Multidimensional Quality Metrics) framework with domain expert annotators to assess fluency, adequacy, terminology handling, and domain-specific accuracy across all 60 translations. (5) Analysis stage—correlate translation failure patterns with terminology frequency statistics extracted from LLM training data proxies (e.g., frequency in published Greek corpora), identifying the rarity threshold where failures become categorical rather than continuous.
Production Impact
For engineers building LLM translation systems for specialized or low-resource domains, this paper provides a concrete diagnostic tool: before deploying an LLM translator, measure terminology rarity against your source domain and establish failure thresholds empirically. If you're translating medical, legal, or technical texts in modern languages, you face the same rarity problem at smaller scales—rare chemical names, archaic legal terms, or domain-specific jargon will cause similar degradation. In production, you'd need to implement fallback mechanisms for rare terms (dictionary lookup, post-editing, human review gates) rather than trusting end-to-end LLM translation. The paper also justifies the added cost of expert human evaluation over automated metrics alone; for high-stakes translation (medical texts, legal documents, scientific work), BLEU scores mask catastrophic failures that human evaluation catches. The trade-off is clear: pure LLM translation is unreliable for terminology-dense domains, requiring either (a) fine-tuning on domain corpora to increase rare term frequency in the effective training distribution, (b) hybrid pipelines with term replacement, or (c) human-in-the-loop validation.
Limitations and When Not to Use This
The evaluation is restricted to Ancient Greek medical prose from a single author (Galen), which limits generalizability to other low-resource ancient languages or modern specialized domains—the findings may not hold equally for Ancient Egyptian, Linear B, or Sumerian texts. The paper relies on reference translations for one of two source works (On Mixtures) and lacks references entirely for the other, making reference-free evaluation methodology crucial but potentially noisier than traditional MT evaluation with gold-standard references. The three commercial LLMs evaluated (Claude, Gemini, ChatGPT) are all large closed-source models trained on broad web corpora; the findings may not apply to open-source models, domain-specific fine-tuned variants, or future architectures with different training procedures. The paper does not clearly distinguish whether failures stem from insufficient training data exposure to rare terms, fundamental limitations of the transformer architecture in handling out-of-distribution vocabulary, or suboptimal prompting/decoding strategies—follow-up work would need to ablate these factors systematically.
Research Context
This work builds on decades of machine translation evaluation research (BLEU, METEOR, MQM frameworks) and applies it to an underexplored frontier: LLM translation of ancient languages. It extends prior low-resource MT work by testing commercial LLMs rather than trainable sequence-to-sequence models, revealing that scale and general knowledge alone do not solve the rare-terminology problem. The paper contributes to the broader safety and robustness literature around LLMs by identifying a concrete failure mode—catastrophic degradation on out-of-distribution inputs—and showing it's predictable and measurable. This opens a research direction into how to make LLMs robust to terminology rarity: through better training data curation, architectural changes, or retrieval-augmented generation approaches that ground outputs in domain lexicons.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::