Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues
| Authors | Yu Wang et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.20079 |
| Download | |
| Categories | cs.CL |
Abstract
We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.
Engineering Breakdown
Plain English
This paper investigates how linguistic features in real-world dialogue can predict a listener's moment-by-moment understanding during explanations. The researchers analyzed the MUNDEX corpus of face-to-face board game explanations and tested three specific signals: information value (measured via surprisal), syntactic complexity of speaker utterances, and variation in listener gaze behavior. They found that individual cues correlate with four listener states: Understanding, Partial Understanding, Non-Understanding, and presumably a fourth category (abstract cuts off). The work bridges NLP and cognitive science by operationalizing abstract understanding as a measurable prediction task grounded in multimodal dialogue data.
Core Technical Contribution
The core novelty is operationalizing real-time listener understanding prediction as a classification task using linguistically-grounded features rather than end-to-end deep learning. The authors move beyond binary comprehension assessment to a four-way classification of listener states, incorporating both speaker signals (surprisal, syntactic complexity) and listener signals (interactive gaze). This multi-modal, feature-based approach directly connects information-theoretic and psycholinguistic concepts (surprisal from information theory, gaze as cognitive load indicator) to a practical NLP prediction task. The contribution is methodological rather than architectural—showing that carefully selected linguistic cues have predictive power for understanding without requiring large annotated datasets or complex neural models.
How It Works
The system extracts features from dialogue at the utterance and gesture level: (1) surprisal scores quantify how predictable each word is in context, using a language model to measure information value; (2) syntactic complexity metrics characterize the structural difficulty of speaker utterances; (3) gaze variation tracks listener eye movement patterns as a proxy for cognitive load. These features are computed on a moment-by-moment basis synchronized with dialogue turns. A classifier (likely logistic regression or similar given the statistical analysis emphasis in the abstract) takes these three feature sets and predicts one of four listener understanding states. The MUNDEX corpus provides ground truth labels for listener states, presumably annotated by human raters observing the dialogues, allowing supervised training and evaluation of which cues matter most for each state.
Production Impact
For production dialogue systems and tutoring applications, this approach enables real-time adaptation of explanation strategies without requiring expensive video analysis at inference time. A dialogue system could compute surprisal and syntactic complexity on-the-fly for its own utterances, then use a lightweight classifier to estimate user understanding and trigger clarifications or simplifications accordingly. The gaze component is a limitation in many production settings (most text-based or voice interactions lack video), but the surprisal and complexity features are cheaply computed using existing language models. The trade-off is that this approach requires task-specific annotation of understanding states during system development; you cannot simply train end-to-end without labeled dialogue data. Latency is minimal—feature extraction and inference happen in milliseconds—making real-time dialogue adaptation feasible.
Limitations and When Not to Use This
The approach is heavily constrained to face-to-face dialogue scenarios with video, making it difficult to transfer to text-only chats, voice assistants, or asynchronous communication where gaze is unavailable. The four-way understanding classification may not generalize to different domains (academic lectures, customer support, technical documentation) where the linguistic patterns and cognitive load drivers differ substantially. The paper does not appear to provide an end-to-end dialogue system that closes the loop—adapting explanations based on predicted understanding and measuring whether adaptation actually improves outcomes. Dataset and annotation costs are substantial: the MUNDEX corpus required human observation and labeling of real dialogues, creating a barrier to scaling. Finally, the statistical feature-based approach may miss non-linear interactions between cues or long-range dialogue context that deep learning could capture, leaving room for hybrid approaches that combine interpretable features with neural components.
Research Context
This work sits at the intersection of psycholinguistics, cognitive science, and NLP, building on decades of research linking surprisal and gaze to cognitive load. It extends prior work on understanding detection in dialogue (often framed as clarification question prediction or dialogue breakdown detection) by explicitly modeling listener states and grounding predictions in linguistically interpretable features. The use of surprisal as a feature connects to information-theoretic approaches in language understanding (work by Levy, Hale, and others). The MUNDEX corpus itself is likely a contribution—a specialized dataset for studying real-world dialogue with understanding annotations—that enables future work on understanding-aware dialogue systems. This opens a research direction toward adaptive explanatory dialogue that continuously monitors comprehension and adjusts complexity in real time.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::