Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
| Authors | Lester James V. Miranda et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.11290 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.
Engineering Breakdown
Plain English
This paper addresses a critical problem in synthetic data generation for multilingual AI: most teams simply use the largest available language model as a teacher to create training data for smaller student models, without checking if that teacher is actually good at non-English tasks. The authors systematically evaluated 10 different language models across 6 typologically diverse languages, generating over 1.4 million supervised finetuning examples and training 240 student models to understand which teacher models produce the highest-quality synthetic data. They introduced a metric called Polyglot Score that combines intrinsic data quality measures with actual downstream student performance, revealing that the largest model is often not the best teacher for multilingual tasks. The key finding is that teacher model selection matters significantly for student performance, and choosing the right teacher—not just the biggest one—can substantially improve outcomes across non-English languages.
Core Technical Contribution
The core contribution is a systematic framework for selecting effective multilingual teacher models based on measured data quality and student performance correlation, rather than defaulting to the largest available model. The authors introduce Polyglot Score, a new evaluation metric that bridges the gap between intrinsic synthetic data quality (measured at generation time) and extrinsic student model performance (measured after finetuning and evaluation). This is novel because prior work on synthetic data generation and knowledge distillation typically assumed larger teachers are better without empirical validation across diverse languages. The paper also provides the first large-scale empirical study of teacher-student dynamics in multilingual SFT, evaluating 10 teacher models systematically rather than making ad hoc selections.
How It Works
The technical pipeline operates as follows: (1) Ten language models of varying sizes and architectures are selected as potential teachers. (2) Each teacher generates supervised finetuning data across six typologically diverse languages—this means languages from different language families (e.g., Romance, Slavic, East Asian) to test generalization. (3) For each teacher-language pair, the authors measure intrinsic data quality metrics, which could include fluency scores, semantic consistency, and adherence to task specifications—these are properties of the generated data itself without training on it. (4) Multiple student models are trained on the synthetic data from each teacher. (5) Students are evaluated on downstream tasks in their respective languages, producing extrinsic performance scores. (6) Polyglot Score correlates these intrinsic and extrinsic measurements to identify which teachers produce data that translates to actual student performance gains. The authors likely use statistical correlation or regression to quantify which teacher characteristics (model size, multilingual pretraining, etc.) predict student success.
Production Impact
For teams building multilingual AI systems, this work immediately changes the teacher selection process from a binary choice (pick the largest model) to a data-driven decision based on measured performance. If you're currently generating synthetic training data for 50 languages using GPT-4 or Llama-3 as the default teacher, this paper suggests you should benchmark a handful of candidate teachers on a representative sample of your tasks first—measuring both data quality and downstream accuracy on your actual student models. The concrete production benefit is improved student model performance across non-English languages without increasing compute costs (you're still training the same student models, just on better synthetic data). However, there are trade-offs: you now need to run an upfront evaluation phase with multiple teachers, which adds latency to your development cycle. You'll also need robust data quality measurement infrastructure and enough compute to train evaluation student models across different teacher-student combinations, which could take weeks depending on scale.
Limitations and When Not to Use This
The paper evaluates only 6 languages and 10 teacher models, which may not capture the complexity of real-world multilingual systems that need to support 50+ languages with different resource availability. The study assumes that synthetic data quality metrics can be reliably measured intrinsically without training, but this assumption may break down for complex reasoning tasks where data quality is task-specific rather than universal. The paper does not deeply explore computational costs: selecting the optimal teacher requires training multiple student models for evaluation, which could be prohibitively expensive for teams with limited compute budgets. Additionally, the work assumes teachers and students use similar architectures and vocabularies; the findings may not transfer to scenarios where you're distilling from a 70B model to a 7B model with completely different tokenizers or architectural choices. Finally, the paper likely does not address the question of whether findings generalize to newer model families or significantly different downstream tasks beyond the ones evaluated in the study.
Research Context
This work builds on the knowledge distillation literature, which has long studied how to transfer knowledge from large teacher models to smaller student models, but extends it to the multilingual and synthetic-data-generation setting. It relates to recent work on synthetic data quality (e.g., using LLMs to generate training data) and multilingual model evaluation, but adds the missing empirical bridge between data quality and student performance. The paper likely cites work on language model evaluation (like MMLU or multilingual variants) and prior studies on teacher-student model pairs, but distinguishes itself by systematically varying the teacher while holding the student and evaluation setup constant. This opens up research directions in automated teacher selection, dynamic teacher routing (using different teachers for different languages), and understanding what linguistic capabilities in a teacher translate to better student performance on downstream tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::