RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 39 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ivan Bondarenko et al. |
| Year | 2026 |
| HF Upvotes | 39 |
| arXiv | 2605.04523 |
| Download | |
| Code | https://github.com/RaguTeam/ragu_mtrag_semeval |
Abstract
We present our winning system for TaskB (generation with reference passages) in SemEval-2026 Task8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval
Engineering Breakdown
Plain English
This paper describes a winning system for a multi-turn dialogue task that uses seven different LLMs with varying prompting strategies, then has a GPT-4o-mini judge pick the best response for each instance. The ensemble achieved a harmonic mean score of 0.7827, beating the strongest baseline (0.6390) and winning 1st place out of 26 teams. The authors also released Meno-Lite-0.1, a 7B fine-tuned model that offers good performance at lower cost.
Key Engineering Insight
Heterogeneous ensembles with a learned judge outperform single-model approaches significantly—but only when you combine models of different sizes, families, and prompting strategies. Any single model in their ensemble underperformed the ensemble, showing that orchestration diversity is essential, not just quantity.
Why It Matters for Engineers
For production systems serving multi-turn dialogue, this validates a practical pattern: instead of optimizing one large model, route candidate generations through multiple models and use a fast judge to select winners. This can improve quality without proportional cost increases—their 7B adapted model shows you can achieve strong cost-performance tradeoffs by domain-tuning smaller models rather than scaling up.
Research Context
Ensemble methods for LLMs have existed, but this work advances the specific problem of faithful multi-turn response generation where reference passages must be correctly used. It shifts from 'use one good model' to 'orchestrate diverse models with intelligent selection,' and the public release of Meno-Lite-0.1 enables other teams to build on domain-adapted smaller models rather than relying solely on API-based giants.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::