Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation
| Authors | Yifeng Liu et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.13045 |
| Download | |
| Categories | cs.CL |
Abstract
Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.
Engineering Breakdown
Plain English
This paper presents WALAR, a reinforcement learning method that improves LLM translation quality for low-resource languages using only monolingual text, without requiring expensive parallel data. The key innovation is identifying and addressing 'holes' (failure modes) in existing multilingual quality estimation models that RL would normally amplify during training. The approach maintains performance on high-resource language pairs while enabling the model to learn from unlabeled monolingual data across hundreds of low-resource languages. This is significant because parallel corpora are scarce or nonexistent for most language pairs globally, making this approach highly practical for real-world deployment.
Core Technical Contribution
The core insight is that source-based multilingual quality estimation (QE) models have systematic failure modes that, when used as reward signals in reinforcement learning, get amplified rather than corrected. WALAR introduces a mechanism to identify these 'holes' in QE models and applies RL in a way that avoids or mitigates them, enabling training on monolingual data only. This differs fundamentally from prior post-training methods that rely on high-quality parallel data or assume QE models are sufficiently reliable. The novelty lies in making the RL-QE pipeline robust enough to work with imperfect quality signals while scaling to hundreds of low-resource language pairs simultaneously.
How It Works
The system begins with a pretrained multilingual LLM and a source-based quality estimation model (trained on parallel data for high-resource languages). During RL training, the model generates translations for monolingual source text, and the QE model scores them. Rather than directly using these scores as rewards, WALAR identifies failure modes in the QE model—cases where it gives high scores to genuinely poor translations or vice versa. The algorithm then either masks out these unreliable signals or applies a correction mechanism to prevent the RL policy from exploiting QE weaknesses. The policy gradient updates proceed using the corrected reward signal, iteratively improving translation quality on low-resource pairs while a separate loss term preserves performance on high-resource language pairs. The entire process operates on monolingual corpora, requiring no parallel data during the RL phase.
Production Impact
For production systems serving global users, this solves a critical business problem: enabling machine translation for the long tail of low-resource languages where parallel training data simply doesn't exist. Rather than accepting poor translation quality or investing heavily in data collection, engineers can now leverage monolingual text (which is abundant for almost every language via web crawls) to bootstrap translation capabilities. The approach reduces infrastructure costs by eliminating the need to source, clean, and align parallel corpora for hundreds of language pairs. However, teams must manage the QE model quality carefully—it becomes a critical component whose failures directly propagate to translation quality—and should implement monitoring to detect when QE model assumptions break down on new domains or language families. The RL training loop adds computational overhead compared to supervised fine-tuning, requiring careful tuning of RL hyperparameters and longer training times.
Limitations and When Not to Use This
The paper's effectiveness depends critically on the quality of the source-based QE model, which is only trained on high-resource language pairs; there's no guarantee these quality signals transfer reliably to languages with different morphology, syntax, or script systems. The approach assumes monolingual data is available and representative of the translation domain, but in practice, web-crawled text may have significant domain mismatch with actual translation use cases. The mechanism for identifying and correcting QE 'holes' is not fully detailed in the abstract, leaving open questions about whether this generalization works across diverse language families or only within language clusters. The paper does not discuss how to handle cases where the source-based QE model completely fails on a language pair, or provide empirical data on which language families benefit most from this approach.
Research Context
This work builds on the intersection of quality estimation research, reinforcement learning for NLP, and multilingual LLM adaptation. It directly addresses limitations of prior post-training methods (like supervised fine-tuning with parallel data) that have difficulty scaling to the 7,000+ human languages. The research connects to broader trends in using monolingual data for translation improvement and extends techniques like RLHF to the multilingual setting where reward model reliability is harder to ensure. This opens research directions into making RL more robust when reward signals are imperfect or domain-shifted, and suggests that QE model analysis could unlock new approaches to low-resource NLP tasks beyond translation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::