Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization
| Authors | Dipto Sumit et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03192 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
We study multiteacher knowledge distillation for low resource abstractive summarization from a reliability aware perspective. We introduce EWAD (Entropy Weighted Agreement Aware Distillation), a token level mechanism that routes supervision between teacher distillation and gold supervision based on inter teacher agreement, and CPDP (Capacity Proportional Divergence Preservation), a geometric constraint on the student position relative to heterogeneous teachers. Across two Bangla datasets, 13 BanglaT5 ablations, and eight Qwen2.5 experiments, we find that logit level KD provides the most reliable gains, while more complex distillation improves semantic similarity for short summaries but degrades longer outputs. Cross lingual pseudo label KD across ten languages retains 71-122 percent of teacher ROUGE L at 3.2x compression. A human validated multi judge LLM evaluation further reveals calibration bias in single judge pipelines. Overall, our results show that reliability aware distillation helps characterize when multi teacher supervision improves summarization and when data scaling outweighs loss engineering.
Engineering Breakdown
Plain English
This paper tackles low-resource abstractive summarization by studying how to effectively distill knowledge from multiple teacher models into a single student model, with a focus on reliability. The authors propose two novel mechanisms: EWAD (Entropy Weighted Agreement Aware Distillation) which intelligently routes training signals between teacher distillation and gold labels based on how much the teachers agree, and CPDP (Capacity Proportional Divergence Preservation) which constrains how the student model positions itself relative to heterogeneous teachers. Testing on Bangla datasets with BanglaT5 and Qwen2.5 models, they found that simple logit-level knowledge distillation provides the most reliable improvements, while more complex methods help with semantic similarity on short summaries but actually hurt performance on longer outputs. Cross-lingual pseudo-labeling across ten languages retained 71-122% of the teacher model's performance, suggesting the approach generalizes well beyond the training language.
Core Technical Contribution
The paper introduces EWAD, a token-level distillation routing mechanism that dynamically selects between teacher supervision and gold-label supervision based on inter-teacher agreement scores weighted by entropy. This addresses a real problem in multi-teacher distillation: not all teachers are equally reliable on all tokens, so blindly mixing their signals can hurt student performance. The second contribution, CPDP, is a geometric constraint that prevents the student from collapsing toward any single teacher or overfitting to one teacher's idiosyncrasies, treating the multi-teacher setup as a heterogeneous ensemble rather than a single authority. These mechanisms are specifically designed for low-resource settings where gold labels are scarce and teacher quality varies, making the distillation process more principled than standard approaches that treat all supervision equally.
How It Works
The system operates at the token level during training. For each token position in the generated summary, the model first computes agreement between teacher models on their output distributions (typically measured via entropy of agreement scores). EWAD then uses this agreement signal to weight how much to trust teacher distillation versus gold labels—high agreement means teachers probably got it right, so use their signal; low agreement means they're uncertain, so fall back to gold labels if available. CPDP operates as a regularization constraint on the student's learned representations, preventing it from clustering too tightly around any single teacher's output space. During inference, the student model generates summaries using standard decoding (beam search or similar). The training loss combines three components: the standard cross-entropy loss from gold labels, the distillation loss from teacher logits (weighted by EWAD agreement scores), and the CPDP geometric penalty that keeps the student positioned in a balanced zone relative to all teachers' latent representations.
Production Impact
For teams building summarization systems in low-resource languages or domains, this approach directly reduces the need for expensive gold-labeled training data by making better use of weak supervision from multiple teacher models. You'd implement this by running inference on multiple pre-trained teacher models (which could be different architectures or checkpoints), computing token-level agreement statistics, and using those to weight your training signal—this adds ~10-15% overhead during training but requires no changes to inference. The key production insight is that EWAD acts as a quality filter: noisy teacher predictions are automatically de-weighted, so your student model becomes more robust than if you'd just averaged teacher outputs. Trade-offs include increased training complexity (you need multiple teachers in memory or sequential teacher inference), longer training time due to agreement computation, and the requirement to validate that your teacher ensemble is actually diverse enough to benefit from agreement-based routing—if all teachers make the same mistakes, EWAD provides no benefit.
Limitations and When Not to Use This
The paper's results show a critical limitation: while logit-level KD provides consistent gains, more sophisticated distillation (including EWAD/CPDP) degrades performance on longer summaries, suggesting the approach may not generalize well to longer-form generation tasks common in production. The method assumes you have multiple teacher models available, which adds significant computational and memory cost; for single-teacher or budget-constrained scenarios, simpler approaches likely suffice. Evaluation is limited to Bangla datasets and models, leaving unclear how well this transfers to high-resource languages like English or to very different architectures beyond T5-style models. The cross-lingual results (71-122% retention) show concerning variance—the 71% lower bound represents significant degradation, suggesting the approach's reliability breaks down in truly distant language pairs or when teacher diversity is too high.
Research Context
This work builds on decades of knowledge distillation research (Hinton et al., 2015) and recent multi-teacher distillation work, advancing the field by explicitly modeling teacher reliability rather than treating all teachers as equally authoritative. It connects to the broader trend of distillation in low-resource NLP, where labeled data scarcity makes teacher supervision critical. The paper contributes to understanding when and why complex distillation mechanisms help (semantic metrics improve) versus hurt (longer sequences degrade), filling a gap in the literature around distillation robustness. The cross-lingual experiments position this as relevant to the expanding work on cross-lingual knowledge transfer and multilingual model compression, especially important as attention shifts toward non-English languages.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::