From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring
| Authors | Minh Hoang Nguyen et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.06424 |
| Download | |
| Categories | cs.CL |
Abstract
Large language models (LLMs) have recently reshaped Automated Essay Scoring (AES), yet prior studies typically examine individual techniques in isolation, limiting understanding of their relative merits for English as a Second Language (L2) writing. To bridge this gap, we presents a comprehensive comparison of major LLM-based AES paradigms on IELTS Writing Task~2. On this unified benchmark, we evaluate four approaches: (i) encoder-based classification fine-tuning, (ii) zero- and few-shot prompting, (iii) instruction tuning and Retrieval-Augmented Generation (RAG), and (iv) Supervised Fine-Tuning combined with Direct Preference Optimization (DPO) and RAG. Our results reveal clear accuracy-cost-robustness trade-offs across methods, the best configuration, integrating k-SFT and RAG, achieves the strongest overall results with F1-Score 93%. This study offers the first unified empirical comparison of modern LLM-based AES strategies for English L2, promising potential in auto-grading writing tasks. Code is public at https://github.com/MinhNguyenDS/LLM_AES-EnL2
Engineering Breakdown
Plain English
This paper systematically compares four different LLM-based approaches for automated essay scoring on IELTS Writing Task 2, a standardized English proficiency test. The authors evaluated encoder-based fine-tuning, zero/few-shot prompting, instruction tuning with RAG, and a more sophisticated method combining Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and RAG on a unified benchmark. Their key finding is that the combination of k-SFT and RAG achieves the best performance, but this comes with clear trade-offs between accuracy, computational cost, and robustness to adversarial inputs. This is the first comprehensive head-to-head comparison of these major paradigms for L2 writing assessment, filling a gap where prior work typically studied techniques in isolation.
Core Technical Contribution
The core contribution is a rigorous comparative framework that quantifies accuracy-cost-robustness trade-offs across four distinct LLM-based AES paradigms on a common benchmark. Rather than proposing a novel algorithm, the authors' novelty lies in systematically evaluating encoder-based classification, prompting-based approaches, retrieval-augmented generation, and preference optimization methods side-by-side with consistent metrics. They demonstrate that the Supervised Fine-Tuning + DPO + RAG combination outperforms simpler baselines, revealing that this three-component approach is necessary for optimal performance. This comparative methodology is itself the contribution—it establishes what works best for L2 essay scoring and under what constraints, enabling practitioners to make informed trade-off decisions.
How It Works
The paper evaluates four distinct pipelines. The first uses an encoder-only model (like BERT) fine-tuned as a multi-class classifier on labeled IELTS essays to directly predict scores. The second uses prompt-based approaches with LLMs, either zero-shot ("score this essay") or few-shot (providing examples before asking for a score). The third approach combines instruction tuning—where the model is fine-tuned on task-specific instructions—with Retrieval-Augmented Generation (RAG), which retrieves relevant reference essays from a database to ground the scoring decision. The fourth and best-performing approach applies Supervised Fine-Tuning (SFT) where the model learns to generate score explanations, then uses Direct Preference Optimization (DPO) to align outputs with human preference rankings, combined with RAG for retrieval. Each pipeline produces essay scores, but they differ in required training data, inference latency, and robustness to distribution shift.
Production Impact
For teams building automated grading systems, this paper provides a concrete decision tree: if you have limited compute and need immediate deployment, encoder-based classification gives reasonable accuracy with minimal overhead; if you can afford API calls to frontier LLMs, few-shot prompting requires no training but may be less consistent; if you have labeled essays and want robustness, the SFT+DPO+RAG approach delivers the best accuracy but requires maintaining a retrieval index and running inference through multiple stages. The production trade-off is clear: the best approach requires (1) 10,000+ labeled essays for SFT and DPO, (2) a vector database for RAG with essay embeddings, (3) ~2-3x the inference latency of a simple classifier, and (4) careful handling of retrieval quality. Organizations targeting high-stakes assessments (where accuracy matters most) should invest in the full pipeline; those prioritizing speed and minimal maintenance should use few-shot prompting despite lower consistency. The paper also surfaces robustness concerns—adversarial attacks or out-of-distribution writing patterns—which affect which approach to choose depending on your threat model.
Limitations and When Not to Use This
The paper focuses exclusively on IELTS Writing Task 2, a specific English proficiency test format with defined rubrics and score bands (0-9); results may not generalize to other essay types like creative writing, coding assessments, or non-English L2 assessments with different rubric structures. The abstract indicates the best method uses k-SFT (unclear if k refers to k-shot or a specific variant) and RAG, but the paper doesn't clarify failure modes—for instance, what happens when the retrieval database contains off-topic or low-quality reference essays, or when a student's writing style is radically different from the training distribution. The comparison assumes access to sufficient labeled IELTS data; for lower-resource languages or custom rubrics, the relative performance ranking may shift dramatically. Additionally, the paper doesn't address real-time scoring latency in detail, which matters for live exam administration, nor does it discuss how these methods handle edge cases like extremely short essays, non-native script mixing, or deliberate obfuscation.
Research Context
This work builds directly on recent LLM breakthroughs in NLP where models like GPT-3.5/4, Claude, and instruction-tuned variants showed surprising zero-shot abilities. Prior AES research focused on individual techniques—some papers explored neural classifiers, others studied prompt engineering, others examined retrieval methods—but lacked systematic comparison. The IELTS Writing Task 2 benchmark has become a standard testbed for L2 assessment because it has clear rubrics, publicly available samples, and real-world importance (IELTS scores gate university admissions). This comparative study fills a methodological gap by establishing a unified evaluation framework, similar in spirit to how benchmark papers (ImageNet, SuperGLUE) have driven progress elsewhere. The work opens research directions around adversarial robustness for scoring systems, the role of retrieval quality in preference optimization, and whether these methods transfer to other assessment domains (medical writing, professional certifications, code documentation).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::