ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
| Authors | Hui Sun et al. |
| Year | 2026 |
| HF Upvotes | 52 |
| arXiv | 2604.03922 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a circular dependency. Our key insight is that we need not determine test correctness at all: test votes should rank, not merely count. What matters is not how many codes pass a test, but whether the test can distinguish correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose ACES~(AUC ConsistEncy Scoring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@k on multiple code generation benchmarks.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in using LLMs to generate and test code: the tests themselves may be wrong, creating a circular dependency where you need correct code to validate tests, but need valid tests to select correct code. The authors propose ACES (Leave-One-Out AUC Consistency), which breaks this circularity by using leave-one-out evaluation to rank tests by their ability to distinguish correct from incorrect code, rather than counting how many codes pass each test. Instead of trying to determine which tests are correct, the method measures test quality by whether a test's pass/fail pattern aligns with code rankings derived from all other tests. This approach enables more reliable selection of LLM-generated code without requiring ground truth about test correctness.
Core Technical Contribution
The core novelty is reframing test reliability assessment as a ranking problem rather than a classification problem. Rather than determining absolute test correctness (impossible without circular reasoning), ACES measures whether each test can discriminate good code from bad code by comparing a test's verdicts against a ranking derived from the aggregate signal of other tests. This is implemented through leave-one-out cross-validation on test judgments: for each held-out test, compute code rankings from remaining tests, then measure whether the held-out test's pass/fail pattern correlates with those rankings using AUC consistency. This elegant reformulation eliminates the need for ground truth labels while still providing a principled way to weight and filter unreliable tests.
How It Works
The algorithm operates in iterations over the set of candidate tests. For each test i to be evaluated: (1) exclude test i and compute aggregate scores for all code candidates based on their performance on the remaining tests—this gives a ranking of codes from best to worst without test i's influence; (2) examine test i's pass/fail verdicts and measure how well they agree with the ranking computed in step 1 using area under the curve (AUC) or similar ranking correlation metrics; (3) assign test i a quality score based on this agreement—high AUC means the test reliably separates good code from bad code, low AUC means the test is noisy or incorrect. Once all tests have quality scores, the method can either weight code selection votes by test quality scores (higher-quality tests count more) or filter out tests below a threshold. The final code ranking aggregates weighted votes across all tests, with test i's contribution proportional to its leave-one-out AUC consistency score.
Production Impact
In a production code generation pipeline, this directly improves the reliability of code selection without requiring expensive human review of all generated tests. Instead of treating all test verdicts equally or using hand-tuned heuristics, engineers can automatically identify which generated tests are trustworthy—critical when test generation itself is error-prone. This reduces the false acceptance rate of incorrect code that happens to pass bad tests, while improving recall on correct code that may fail spurious tests. Implementation is straightforward: replace uniform test weighting with leave-one-out AUC-based weighting in the aggregation step, adding O(n²) leave-one-out passes over n tests—acceptable for typical code generation workloads where test counts are in the tens or low hundreds. The approach integrates cleanly into existing generate-and-test pipelines without architectural changes, and provides explainability (engineers can inspect which tests the system deems high-quality).
Limitations and When Not to Use This
The method assumes that most tests are at least partially correlated with true code correctness; if test verdicts are near-random or all tests are equally broken, leave-one-out AUC consistency cannot distinguish signal from noise. It also assumes sufficient code diversity in the candidate set—if all generated codes are similarly correct or incorrect, no test can achieve high discriminative power. The approach requires O(n²m) evaluations where n is the number of tests and m is the number of code candidates, which becomes expensive for very large candidate pools or when test execution itself is slow. Finally, the method is inherently post-hoc: it requires generating multiple code candidates and multiple test candidates before evaluation begins, so it cannot be used to guide the generation process itself in real-time. Edge cases include test sets with strong correlations (tests that measure the same property) where leave-one-out may not fully decorrelate signal.
Research Context
This work builds on the growing literature around code generation evaluation, particularly research on test-based selection of LLM outputs (e.g., execution-guided decoding, test-driven program synthesis). It addresses a known problem with self-generated test suites: tests can be as buggy as the code they're meant to evaluate. The paper extends ideas from robust statistics and anomaly detection (using relative rankings rather than absolute judgments) into the code generation domain. It complements concurrent work on improving test generation quality and relates to broader questions in AI alignment about how to evaluate systems when ground truth is unavailable—a relevant concern for code generation where test correctness itself depends on human intent. The leave-one-out evaluation paradigm is standard in ML (used in cross-validation), but its application to test reliability assessment in code generation appears novel and opens avenues for similar consistency-checking approaches in other generate-and-evaluate pipelines.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::