Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
| Authors | Evangelia Christakopoulou et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.23234 |
| Download | |
| Categories | cs.IR, cs.AI, cs.LG |
Abstract
Large-scale commercial search systems optimize for relevance to drive successful sessions that help users find what they are looking for. To maximize relevance, we leverage two complementary objectives: behavioral relevance (results users tend to click or download) and textual relevance (a result's semantic fit to the query). A persistent challenge is the scarcity of expert-provided textual relevance labels relative to abundant behavioral relevance labels. We first address this by systematically evaluating LLM configurations, finding that a specialized, fine-tuned model significantly outperforms a much larger pre-trained one in providing highly relevant labels. Using this optimal model as a force multiplier, we generate millions of textual relevance labels to overcome the data scarcity. We show that augmenting our production ranker with these textual relevance labels leads to a significant outward shift of the Pareto frontier: offline NDCG improves for behavioral relevance while simultaneously increasing for textual relevance. These offline gains were validated by a worldwide A/B test on the App Store ranker, which demonstrated a statistically significant +0.24% increase in conversion rate, with the most substantial performance gains occurring in tail queries, where the new textual relevance labels provide a robust signal in the absence of reliable behavioral relevance labels.
Engineering Breakdown
Plain English
This paper tackles a fundamental problem in large-scale app store search: there's abundant data about what users actually click (behavioral relevance), but very few expert-labeled examples of which results semantically match queries (textual relevance). The authors discovered that a smaller, fine-tuned LLM dramatically outperforms a much larger pre-trained model at generating these textual relevance labels. By using this optimized model to generate millions of labels at scale, they effectively solved the label scarcity problem and improved search ranking for production systems serving millions of users.
Core Technical Contribution
The core innovation is demonstrating that specialized, fine-tuned smaller LLMs can be more effective than large foundational models for generating domain-specific relevance judgments in commercial search. Rather than using off-the-shelf LLMs, the authors systematically evaluated configurations and found that targeted fine-tuning on app store data dramatically improved label quality. This enables treating the optimized LLM as a 'force multiplier' to bootstrap millions of high-quality training examples from scratch, directly addressing the label scarcity bottleneck that typically limits relevance ranking systems.
How It Works
The system operates by combining two complementary relevance signals: behavioral relevance (click-through and download patterns that users naturally produce) and textual relevance (semantic alignment between query and app description/metadata). The authors treat textual relevance as a learning problem: they first curate a smaller set of expert-labeled examples, then fine-tune an LLM specifically on app store ranking tasks. This tuned model is evaluated against multiple baselines to confirm it produces higher-quality judgments than larger pre-trained alternatives. Once validated, the optimized LLM generates millions of synthetic textual relevance labels at scale. These generated labels are then used as training data for the ranking model, which learns to optimize both behavioral signals (from user interactions) and the newly-labeled textual signals. The ranking system combines these signals during inference to surface apps that users both click on and find semantically relevant to their query.
Production Impact
For teams running production search systems (especially in mobile app stores, e-commerce, or marketplaces), this approach directly addresses the expensive bottleneck of manual relevance labeling. Instead of hiring raters or waiting months for labels, engineers can fine-tune a small LLM on domain data and generate millions of labels in days at modest computational cost. This is particularly valuable when behavioral data exists but is noisy or sparse—the textual signal provides orthogonal information that improves ranking quality. The trade-off is that LLM-generated labels introduce some noise compared to expert judgment, requiring careful validation on a held-out set; however, the scale advantage (millions vs. thousands of labels) typically compensates. Integration is straightforward: label generation becomes an offline batch process, and the labels feed into standard ranking model training pipelines without architectural changes.
Limitations and When Not to Use This
The approach assumes that fine-tuned smaller LLMs can reliably approximate expert relevance judgment, which may not hold for highly subjective queries or niche domains where the training distribution is thin. The paper doesn't deeply explore failure modes when LLM labels conflict with behavioral signals—in practice, generated labels may introduce systematic biases (e.g., favoring certain app categories) that aren't obvious until after deployment. Computational cost and latency for label generation aren't thoroughly benchmarked, making it unclear how the approach scales to truly massive catalogs (billions of items) or real-time labeling scenarios. The method also requires domain-specific fine-tuning data, so the gains may not transfer to search problems with fundamentally different data characteristics (e.g., scientific papers, legal documents).
Research Context
This work builds on a growing line of research exploring LLMs as data generation engines for training discriminative models, extending techniques like data augmentation and weak supervision into the relevance ranking domain. It connects to prior work on learning-to-rank systems and semantic matching, but with a key insight: smaller, tuned models can outperform larger ones for specific downstream tasks. The paper advances the broader agenda of reducing human annotation burden in search—a long-standing challenge in IR and ML for production systems. This opens future directions around mixing behavioral and synthetic textual signals, validating generated labels at scale, and understanding when and why smaller models outperform larger ones in domain-specific tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::