Beyond Retrieval: A Multitask Benchmark and Model for Code Search
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 21 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Siqiao Xue et al. |
| Year | 2026 |
| HF Upvotes | 21 |
| arXiv | 2605.04615 |
| Download | |
| Code | https://github.com/hq-bench/coreb |
Abstract
Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce CoREB, a contamination-limited, multitask code retrieval and reranking benchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. CoREB is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: \circone code-specialised embeddings dominate code-to-code retrieval ({sim}2{times} over general encoders), yet no single model wins all three tasks; \circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; \circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; \circfour our fine-tuned CoREB-Reranker is the first to achieve consistent gains across all three tasks. The data and model are released.
Engineering Breakdown
Plain English
This paper introduces CoREB, a new benchmark for evaluating code search systems that reflects how production systems actually work—with reranking stages and realistic developer queries—rather than just measuring first-stage retrieval. The authors benchmark 11 embedding models and 5 rerankers across three search types (text-to-code, code-to-text, code-to-code) using data built from LiveCodeBench problems in five languages, addressing known issues like data contamination and poor label quality in existing benchmarks.
Key Engineering Insight
Most code search research optimizes for initial retrieval ranking, but production systems critically depend on reranking stages that refine results after retrieval. This paper shows you can't evaluate real-world code search effectiveness without benchmarking the full pipeline end-to-end.
Why It Matters for Engineers
If you're building a code search product or IDE feature, optimizing only your embedding model misses where most relevance gains happen in practice. This benchmark lets you measure whether your reranker actually improves what developers see, and reveals which embedding-reranker combinations work best for your specific use case—avoiding wasted effort tuning isolated components.
Research Context
Code search benchmarks historically focused on retrieval-only evaluation with shallow relevance labels and contaminated training data. This work advances the field by introducing timed releases (preventing future data leakage), graded relevance judgments (replacing binary labels), and multitask evaluation that mirrors production architectures. It establishes a baseline for measuring the gap between retrieval-only metrics and end-to-end search quality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::