ML-Bench&Guard: Policy-Grounded Multilingual Safety Benchmark and Guardrail for Large Language Models
| Authors | Yunhan Zhao et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2605.00689 |
| Download | |
| Categories | cs.CL, cs.CR |
Abstract
As Large Language Models (LLMs) are increasingly deployed in cross-linguistic contexts, ensuring safety in diverse regulatory and cultural environments has become a critical challenge. However, existing multilingual benchmarks largely rely on general risk taxonomies and machine translation, which confines guardrail models to these predefined categories and hinders their ability to align with region-specific regulations and cultural nuances. To bridge these gaps, we introduce ML-Bench, a policy-grounded multilingual safety benchmark covering 14 languages. ML-Bench is constructed directly from regional regulations, where risk categories and fine-grained rules derived from jurisdiction-specific legal texts are directly used to guide the generation of multilingual safety data, enabling culturally and legally aligned evaluation across languages. Building on ML-Bench, we develop ML-Guard, a Diffusion Large Language Model (dLLM)-based guardrail model that supports multilingual safety judgment and policy-conditioned compliance assessment. ML-Guard has two variants, one 1.5B lightweight model for fast `safe/unsafe' checking and a more capable 7B model for customized compliance checking with detailed explanations. We conduct extensive experiments against 11 strong guardrail baselines across 6 existing multilingual safety benchmarks and our ML-Bench, and show that ML-Guard consistently outperforms prior methods. We hope that ML-Bench and ML-Guard can help advance the development of regulation-aware and culturally aligned multilingual guardrail systems.
Engineering Breakdown
Plain English
This paper introduces ML-Bench, a multilingual safety benchmark for large language models covering 14 languages, designed to handle region-specific regulations and cultural safety requirements. Unlike existing multilingual benchmarks that rely on generic risk taxonomies and machine translation, ML-Bench is constructed directly from jurisdiction-specific legal texts, deriving safety rules and risk categories from actual regional regulations. The benchmark enables guardrail models to align with diverse cultural and regulatory contexts rather than being confined to predefined, culturally-agnostic categories. This addresses a critical gap in LLM deployment: current safety mechanisms fail to account for local legal requirements and cultural norms that vary significantly across different regions.
Core Technical Contribution
The core innovation is grounding multilingual safety data generation directly in regional legal and regulatory documents rather than using generic taxonomies translated to multiple languages. The authors developed a policy-grounded methodology that extracts jurisdiction-specific safety rules and risk categories from legal texts, then uses these extracted rules to guide the generation of culturally and legally appropriate safety test cases. This enables fine-grained alignment between LLM behavior and local regulations—something impossible with machine-translated benchmarks that inherit the safety assumptions of their source language. The approach treats safety as a localization problem requiring legal and cultural grounding, not just linguistic translation.
How It Works
The methodology begins by collecting regional legal and regulatory documents from 14 different jurisdictions, extracting safety-relevant rules and risk categories specific to each region's legal framework. These extracted rules then serve as the foundation for generating multilingual safety prompts and test cases, rather than translating a single source benchmark. The benchmark construction pipeline involves: (1) legal document ingestion and rule extraction per jurisdiction, (2) generation of region-appropriate adversarial prompts using extracted rules as guidance, (3) collection of LLM responses to these prompts, (4) annotation of responses against both extracted rules and cultural context. The resulting benchmark contains diverse safety scenarios in 14 languages, with each language's test cases reflecting actual legal and cultural requirements rather than translated versions of English-centric safety concerns. Guardrail models trained or evaluated on this benchmark can learn region-specific patterns and constraints rather than overfitting to translated generic rules.
Production Impact
For engineers building multilingual LLM systems, this benchmark enables testing against actual regulatory requirements rather than generic safety assumptions, reducing compliance risk when deploying in new regions. A production system using ML-Bench would implement region-aware safety classifiers that route responses through jurisdiction-specific guardrails—for instance, a content safety filter for the US market would differ substantially from one for the EU (GDPR), India (IT rules), or other regions. This means organizations need location-aware safety layers, not a single monolithic safety model, increasing architectural complexity but improving legal compliance posture. The trade-off is higher data annotation costs (per-language per-jurisdiction dataset creation) and more complex production inference pipelines with routing logic, but the upside is significantly reduced regulatory and reputational risk. Integration would require storing regional rule sets, mapping user requests to jurisdictions, and maintaining separate or region-conditioned safety models.
Limitations and When Not to Use This
The paper's approach assumes that safety rules can be cleanly extracted from legal documents and that extracted rules will map cleanly to LLM behavior—in practice, legal language is ambiguous and often requires case-law interpretation that static rule extraction misses. The benchmark covers only 14 languages, which leaves thousands of regional contexts unaddressed; scaling to truly global coverage would require proportionally more legal analysis work. It's unclear how the benchmark handles edge cases where regulations conflict (e.g., EU vs US data privacy rules for multinational companies), and the paper doesn't address how to update benchmarks as regulations evolve—legal frameworks change frequently and benchmarks can become stale. Additionally, the approach may not capture informal or emerging cultural safety concerns that aren't yet codified in formal regulations but are important to local communities.
Research Context
This work builds on recent efforts to extend LLM safety beyond English-centric benchmarks like ToxicBench and StereoSet, which have been criticized for not accounting for multilingual and cross-cultural safety differences. It advances the field by proposing legal grounding as a principled alternative to generic taxonomies, moving safety evaluation from abstract harm categories to concrete compliance requirements. The paper contributes to the growing literature on culturally-aware NLP and contextual safety, which recognizes that harmful content definitions vary across cultures and legal systems. ML-Bench opens a research direction toward jurisdiction-aware LLM deployment, where safety mechanisms are explicitly tied to specific regional requirements rather than attempting a one-size-fits-all approach.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::