Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
| Authors | Indraneil Paul et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2605.00754 |
| Download | |
| Categories | cs.SE, cs.LG |
Abstract
Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.
Engineering Breakdown
Plain English
This paper tackles a gap in code generation research by building reward models (RMs) that can evaluate code across multiple quality dimensions—not just whether it executes correctly. The authors created Themis-CodeRewardBench, a benchmark spanning five preference criteria (correctness, efficiency, readability, style, documentation) and eight programming languages, then evaluated 50+ existing RMs to understand their strengths and weaknesses. They found that current RMs, which mostly focus on execution feedback, perform poorly on the broader quality dimensions that matter in real codebases. This work establishes both a systematic evaluation framework and reveals that the field needs RMs trained on more nuanced code quality signals beyond functional correctness.
Core Technical Contribution
The core novelty is multi-criteria reward modeling for code—moving beyond binary execution feedback to structured evaluation across five distinct preference dimensions simultaneously. The authors compiled Themis-CodeRewardBench as the first large-scale benchmark specifically designed to evaluate code RMs on diverse quality criteria across eight languages, enabling systematic comparison of 50+ models. Unlike prior work that treats code generation as a functional correctness problem, this approach recognizes that production code must satisfy efficiency, readability, style consistency, and documentation standards. The empirical insight that existing general-purpose and math RMs fail catastrophically on code-specific criteria motivates the need for specialized, multi-criteria code reward models.
How It Works
The framework operates in three stages: (1) Benchmark construction—collecting code snippets across eight languages with human annotations on five quality dimensions, creating preference pairs that distinguish good from mediocre solutions; (2) Model evaluation—running 50+ candidate RMs (including code-specific, math, and general LM-based RMs) against the benchmark and measuring how well they correlate with human preference judgments on each criterion; (3) Profiling—analyzing failure modes to understand which RM architectures and training approaches work for which criteria. The input is code solutions with reference implementations and explicit quality judgments; the RM outputs scalar preference scores; evaluation uses ranking correlation metrics (likely Spearman or Kendall-τ) to measure agreement with human annotations. The key innovation is separating the scoring into criterion-specific branches rather than forcing a single holistic score, allowing diagnosis of where RMs fail (e.g., code-Llama might score execution highly but miss efficiency).
Production Impact
In production code generation pipelines, this shifts post-training from optimizing only functional correctness to jointly optimizing multiple code quality signals—directly applicable to LLM-powered code assistants, automated testing tools, and CI/CD integration. Teams deploying RLHF or policy gradient methods for code generation can now use multi-criteria RMs to reward not just passing tests but also producing efficient, readable, well-documented code that reduces downstream maintenance costs. The benchmark enables engineers to audit existing RMs before deploying them: if your RM only correlates 0.3 with human readability judgments, you'll generate technically correct but unmaintainable code at scale. Trade-offs include higher labeling cost (multi-criteria annotation is more expensive than binary execution), potential compute overhead during RM inference for scoring multiple criteria, and increased complexity in reward model training pipelines requiring careful multi-task balancing.
Limitations and When Not to Use This
The paper evaluates RMs but doesn't provide the trained models or full details on how to build production-grade code RMs from scratch—practitioners still face engineering work to adapt these insights. The benchmark is limited to code snippets and self-contained problems; it doesn't address reward modeling for long-range dependencies, refactoring decisions, or architectural code quality that spans multiple files and modules. The paper assumes the five chosen criteria capture all relevant code quality dimensions, but production systems may weight criteria differently (safety-critical code prioritizes correctness; performance-sensitive code prioritizes efficiency) requiring context-dependent reward reweighting. Finally, the limited proficiency of current RMs on these criteria suggests the field may need fundamentally different RM architectures rather than just better training data—this work diagnoses the problem but doesn't fully solve it.
Research Context
This work extends the post-training playbook established by RLHF and test-time scaling work (similar to the reasoning scaling trends in math/science domains) but applies it to code with greater rigor than prior code-specific RM papers. It builds on existing code generation benchmarks (HumanEval, CodeChef, etc.) but adds the crucial multi-criteria evaluation dimension that makes RMs useful for realistic code quality, not just pass/fail metrics. The paper parallels recent work on process rewards in reasoning tasks—recognizing that intermediate steps and code properties matter as much as final answers. By establishing Themis-CodeRewardBench as a standard evaluation framework, it opens a research direction toward specialized, criterion-aware RMs that could improve LLM code generation as significantly as RLHF improved language alignment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::