GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
| Authors | Amir Hossein Kargaran et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.12978 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resource scripts. We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR generalization across 100+ Unicode scripts. Our benchmark comprises clean and degraded image variants rendered from real multilingual texts. Images are rendered using fonts from the Google Fonts repository, shaped with HarfBuzz and rasterized with FreeType, supporting both LTR and RTL scripts. Samples of rendered images were manually reviewed to verify correct rendering across all scripts. We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts, and even the strongest frontier models fail to generalize beyond thirty scripts. Performance broadly tracks script-level pretraining coverage, suggesting that current OCR systems rely on language model pretraining as much as on visual recognition. Models confronted with unfamiliar scripts either produce random noise or hallucinate characters from similar scripts they already know. We release the benchmark and pipeline for reproducibility. Pipeline Code: https://github.com/cisnlp/glotocr-bench, Benchmark: https://hf.co/datasets/cis-lmu/glotocr-bench.
Engineering Breakdown
Plain English
GlotOCR Bench is a new benchmark that tests how well OCR models work across 100+ Unicode scripts, addressing a major gap in evaluation coverage. Prior OCR benchmarks focused almost exclusively on a handful of high-resource scripts like English and Chinese, leaving gaps for the other 95% of world's writing systems. The authors created clean and degraded image variants using Google Fonts, HarfBuzz shaping, and FreeType rasterization to render text in diverse scripts. They evaluated both open-weight and proprietary vision-language models and found a sobering result: most state-of-the-art models perform well on fewer than 10 scripts, indicating severe generalization failures across the script diversity landscape.
Core Technical Contribution
The core contribution is GlotOCR Bench itself—a rigorously constructed, multilingual OCR evaluation benchmark spanning 100+ Unicode scripts with systematic rendering pipeline and human verification. Unlike prior OCR datasets that cherry-picked well-supported scripts, this benchmark provides standardized clean and degraded image variants supporting both left-to-right (LTR) and right-to-left (RTL) text directions, enabling fair comparison across script families. The authors demonstrate that this benchmark reveals a critical blind spot in current vision-language models: their OCR capabilities are far narrower than their overall language understanding suggests, with most models failing on 90+ scripts despite strong English/Chinese performance. This sets up the benchmark as a reproducible evaluation artifact that will likely become a standard for measuring OCR generalization, similar to how MMLU became standard for general LLM evaluation.
How It Works
The pipeline starts by sourcing multilingual text samples across 100+ scripts and selecting appropriate fonts from Google Fonts that support each script's characters. The text is shaped using HarfBuzz, which handles complex script-specific rendering rules (ligature formation, diacritics placement, contextual shaping) that vary dramatically across writing systems—this is critical because naive text rendering would produce incorrect glyphs for scripts like Arabic, Devanagari, or Thai. The shaped glyphs are then rasterized into images using FreeType at a standard resolution, with variants created for both clean and degraded conditions (simulating real-world scanned documents with noise, blur, rotation). All rendered images undergo manual review to verify correct glyph formation and legibility, ensuring the benchmark measures model OCR capability rather than rendering artifacts. The evaluation then runs standard vision-language models (both open-weight like LLaVA and proprietary like GPT-4V) on these images and scores character-level or word-level accuracy per script, enabling detailed analysis of which scripts each model handles well.
Production Impact
For production OCR systems, this benchmark immediately exposes a critical reality: if your user base spans scripts beyond the top 5-10, your current vision-language model likely fails silently and often. Teams building multilingual document processing, translation, or accessibility systems should run their models against GlotOCR Bench to identify which scripts lack coverage—this could trigger architectural decisions like fine-tuning on underrepresented scripts, using specialized OCR engines per script family, or investing in better multilingual pretraining. The benchmark's clean/degraded split is also production-relevant because real scanned documents are degraded (low resolution, skew, artifacts), so models need to be tested on both conditions. The systematic rendering pipeline using standard tools (HarfBuzz, FreeType) is reproducible, meaning you can extend it to proprietary fonts or add synthetic degradation specific to your document domain—this is easier than hand-labeling multilingual data. However, adoption requires acknowledging that current models will likely show poor coverage, which may necessitate multiple fallback engines or significant retraining, increasing both latency and infrastructure complexity.
Limitations and When Not to Use This
The benchmark measures character-level OCR recognition but does not address downstream tasks like layout understanding, table extraction, or document structure—many production OCR systems need these, and script-specific layout rules (e.g., vertical vs. horizontal flow) are not evaluated here. The evaluation relies on rendered synthetic text from Google Fonts, which may not match real-world document degradation patterns (ink bleed, fading, handwriting interference, low-quality prints), so models that perform well on GlotOCR may still fail on real scanned documents; validation on authentic multilingual document collections is still needed. The paper evaluates vision-language models as black boxes, but does not deeply analyze why they fail—understanding whether failures stem from insufficient script-specific training data, fundamental architecture limitations (e.g., token embedding misalignment for rare scripts), or optimization choices would unlock targeted fixes. The benchmark's scope is limited to Unicode scripts; it doesn't address historical scripts, synthetic fonts, or non-Latin scripts with irregular character sets, leaving coverage gaps for niche OCR applications.
Research Context
This work builds on a long tradition of OCR benchmarking (Tesseract, ICDAR challenges) but fills a critical gap exposed by the rise of vision-language models: prior benchmarks were dominated by English, Chinese, and a few other high-resource languages, creating an illusion of solved OCR when in reality 90+ scripts were largely ignored. The paper implicitly critiques the narrow evaluation practices in recent vision-language model papers (PaLI, CogVLM, GPT-4V), which typically report results only on English and Chinese text, not on script generalization. GlotOCR Bench is positioned as a universal script evaluation tool, similar in spirit to how FLORES-200 and other multilingual benchmarks expanded language model evaluation beyond English, and it will likely catalyze research into multilingual-aware vision encoders and better script-aware fine-tuning methods. This opens a new research direction: developing OCR models that generalize across the full Unicode space rather than engineering solutions for individual high-resource scripts.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::