Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts
| Authors | Jinsook Lee et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03127 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines (- and -). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.
Engineering Breakdown
Plain English
This paper tackles the problem of automatically annotating tutoring dialogues with pedagogical labels—a task where large language models typically perform poorly without specialized knowledge. Instead of fine-tuning the LLM itself, the authors built a retrieval-augmented generation (RAG) system that uses a lightweight embedding model trained on tutoring data to fetch relevant labeled examples from dialogues. Testing on two real tutoring datasets (TalkMoves and Eedi) with three different LLM backbones, their best approach achieved Cohen's kappa scores of 0.526-0.580 and 0.659-0.743 respectively, roughly doubling the performance of models without retrieval (0.275-0.413 and 0.160-0.410). The key insight is that retrieving utterance-level few-shot examples from the same domain is more effective than using a generic LLM.
Core Technical Contribution
The core novelty is a domain-adapted retrieval pipeline that avoids expensive fine-tuning of generative models by instead fine-tuning only a lightweight embedding model on domain-specific tutoring corpora. Rather than treating the entire dialogue as a single retrieval unit, the authors index at the utterance level, allowing the system to find the most relevant labeled teaching moves for in-context learning. This approach is simpler and cheaper than full model fine-tuning while achieving substantial performance gains—the embedding model acts as a lightweight adapter that learns to recognize semantically similar pedagogical patterns. The ablation study validates that utterance-level indexing is a critical design decision, suggesting that granular retrieval is more effective than coarser dialogue-level indexing for this task.
How It Works
The system works in three stages. First, the authors take a pre-trained embedding model and fine-tune it on tutoring corpora to learn domain-specific representations of pedagogical language—this is much cheaper than fine-tuning a 32B+ parameter LLM. Second, they segment dialogue datasets into individual utterances and index each one in a vector database, associating each utterance with its gold-standard pedagogical label (e.g., 'revoicing', 'eliciting'). Third, at inference time, when the LLM encounters a new tutoring utterance to annotate, it uses the fine-tuned embedding model to retrieve the k most similar labeled utterances from the corpus, then feeds these as few-shot examples into the LLM context window along with the target utterance. The LLM then predicts the label based on both the in-context examples and its pre-training knowledge. The entire retrieval and LLM inference pipeline is evaluated across multiple LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b) to show the approach is model-agnostic.
Production Impact
undefined
Limitations and When Not to Use This
The approach assumes you have a domain-specific labeled corpus of tutoring dialogues available, which is a significant barrier for new domains or languages. The paper evaluates only on two tutoring datasets and doesn't explore how the method performs with smaller labeled corpora or when domain shift occurs (e.g., applying a model trained on high school tutoring to college-level interactions). The absolute Cohen's kappa scores, while improved, remain moderate (0.526-0.743), suggesting the task is still challenging and the system will make annotation errors that require human review in high-stakes educational settings. The ablation study is incomplete—the abstract cuts off before revealing its full findings, so we don't have a complete picture of which components (embedding architecture, retrieval depth, etc.) matter most. The paper also doesn't address how label imbalance, rare pedagogical moves, or out-of-domain utterances are handled by the retrieval system.
Research Context
This work builds on the growing trend of using retrieval to ground LLMs in specialized domains rather than expensive fine-tuning. It extends prior RAG work (like the foundational RAG papers and recent in-context learning research) by applying it specifically to pedagogical dialogue understanding, a high-stakes NLP task where accuracy directly impacts student learning. The paper improves upon baselines on established benchmarks (TalkMoves and Eedi datasets), showing that even modest domain adaptation of the retrieval component can substantially boost LLM performance on structured annotation tasks. This work opens a research direction for other high-stakes domain-specific annotation tasks—medical NLP, legal document classification, scientific paper analysis—where fine-tuning is risky or expensive but domain-adapted retrieval could be highly effective.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::