SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation
| Authors | Deshan Sumanathilaka et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16262 |
| Download | |
| Categories | cs.CL |
Abstract
Recent advances in language models have substantially improved Natural Language Understanding (NLU). Although widely used benchmarks suggest that Large Language Models (LLMs) can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions
Engineering Breakdown
Plain English
This paper tackles word sense disambiguation in narrative contexts—specifically, determining how plausible a particular word meaning is within a short story, which benchmarks haven't adequately tested before. The authors propose an LLM-based framework that uses structured reasoning to score plausibility of homonymous words (words with multiple meanings), combining fine-tuned smaller models with diverse reasoning strategies and few-shot prompting for larger models. The work addresses a gap between theoretical LLM performance on standard benchmarks and practical applicability in real-world storytelling scenarios, where understanding context-dependent word meanings is crucial. While the abstract doesn't provide specific performance numbers, the core finding is that combining parameter-efficient fine-tuning with dynamic prompting improves sense identification accuracy in narrative texts.
Core Technical Contribution
The key novelty is introducing a structured reasoning mechanism for contextual word sense plausibility scoring that explicitly handles narrative coherence—moving beyond token-level classification to story-level semantic understanding. Unlike standard WSD (word sense disambiguation) tasks that treat senses as discrete categories, this framework scores plausibility on a continuous scale reflecting human perception of meaning fitness within narrative flow. The authors demonstrate that smaller fine-tuned models with diverse reasoning strategies can compete with or outperform larger models on this task, suggesting that explicit reasoning structure matters more than raw model scale for narrative understanding. This represents a shift from benchmark-driven evaluation to task formulation that directly measures real-world applicability in creative and narrative contexts.
How It Works
The framework operates in stages: first, it identifies homonymous word candidates in a short story and extracts their surrounding narrative context. For smaller parameter models, the system applies supervised fine-tuning using diverse reasoning strategies—likely chain-of-thought variants that encourage the model to explicitly articulate why each word sense fits or doesn't fit the narrative context. For larger models, the approach uses dynamic few-shot prompting, where examples are selected based on narrative similarity to the current story, and the prompt is structured to elicit plausibility scores (likely on a numerical scale) rather than binary sense labels. The output is a plausibility score per word sense, which can be compared against human annotations to evaluate both accuracy and calibration. The structured reasoning component prevents the model from relying on shallow surface patterns and forces it to construct interpretable justifications for its disambiguation choices.
Production Impact
For teams building narrative understanding systems—book recommendation engines, content moderation, or story generation pipelines—this provides a concrete method to validate semantic coherence without manual human review at scale. The dual-track approach (fine-tuned small models + few-shot large models) offers a practical deployment decision: use fine-tuned models for deterministic, auditable sense scoring when you have domain data and compute budget, or use few-shot prompting with GPT-scale models for flexibility and lower training overhead. The plausibility scoring mechanism enables continuous ranking rather than binary classification, which is more useful in production for ranking or filtering—e.g., selecting the most narrative-coherent interpretation when generating summaries or resolving references. Trade-offs include: fine-tuning requires labeled narrative data (expensive to annotate), structured reasoning increases latency compared to direct classification, and the approach is task-specific (reasoning strategies optimized for narrative may not transfer to technical or scientific text). Integration complexity is moderate—the framework slot into existing NLU pipelines as a post-processing module that refines or validates sense predictions from earlier stages.
Limitations and When Not to Use This
The paper focuses specifically on homonymous words in short stories, which is a narrower setting than open-domain WSD; the reasoning mechanisms and plausibility scoring may not transfer to longer texts, domain-specific language, or non-narrative genres where context is denser and harder to capture. The abstract provides no ablation studies or failure case analysis—it's unclear whether the framework struggles with cultural references, figurative language, or senses that require external world knowledge beyond the narrative. The dynamic few-shot prompting approach for large models depends on having a representative example retrieval mechanism and assumes that narrative similarity is a good proxy for prompt relevance, which may break down in under-represented story types or cross-lingual settings. No information is given about computational overhead or latency impact of the structured reasoning—this could be prohibitive for real-time applications like live text generation or interactive dialogue systems.
Research Context
This work is grounded in the SemEval-2026 Task 5 shared task, which means it's directly advancing the community benchmark for word sense understanding in narrative—prior SemEval tasks (e.g., English Lexical Sample, Semantic Role Labeling) have focused on isolated sentences or argument structure rather than story-level coherence. The paper builds on recent trends in prompting and in-context learning (few-shot methods) alongside parameter-efficient fine-tuning (likely LoRA or adapters), showing that these techniques can be combined for nuanced semantic tasks. It contributes to a growing body of work questioning whether standard benchmarks capture real-world language understanding, joining critiques that motivated the creation of more naturalistic evaluation datasets. The structured reasoning approach connects to broader research on interpretability and chain-of-thought prompting, extending those ideas to the specific domain of sense disambiguation and plausibility judgment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::