Sentiment Analysis of German Sign Language Fairy Tales
| Authors | Fabrizio Nunnari et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16138 |
| Download | |
| Categories | cs.CL, cs.LG |
Abstract
We present a dataset and a model for sentiment analysis of German sign language (DGS) fairy tales. First, we perform sentiment analysis for three levels of valence (negative, neutral, positive) on German fairy tales text segments using four large language models (LLMs) and majority voting, reaching an inter-annotator agreement of 0.781 Krippendorff's alpha. Second, we extract face and body motion features from each corresponding DGS video segment using MediaPipe. Finally, we train an explainable model (based on XGBoost) to predict negative, neutral or positive sentiment from video features. Results show an average balanced accuracy of 0.631. A thorough analysis of the most important features reveal that, in addition to eyebrows and mouth motion on the face, also the motion of hips, elbows, and shoulders considerably contribute in the discrimination of the conveyed sentiment, indicating an equal importance of face and body for sentiment communication in sign language.
Engineering Breakdown
Plain English
This paper tackles sentiment analysis for German Sign Language (DGS) by building a two-stage pipeline: first, they use four large language models with majority voting to label German fairy tale text segments with sentiment (negative, neutral, positive), achieving 0.781 Krippendorff's alpha inter-annotator agreement; second, they extract facial and body motion features from corresponding DGS videos using MediaPipe and train an XGBoost model to predict sentiment from video features alone. The core finding is that an explainable model can predict sentiment from sign language video with 0.631 balanced accuracy, with important signals coming not just from facial expressions (eyebrows, mouth) but also from body motion (hips, elbows). This is the first dataset and model for sentiment analysis specifically grounded in both sign language video and paired text annotations.
Core Technical Contribution
The paper's core novelty is creating a multimodal sentiment annotation pipeline for sign language video paired with text, bridging two typically separate research areas: NLP sentiment analysis and computer vision-based sign language understanding. Rather than building a black-box end-to-end model, the authors deliberately chose an explainable XGBoost approach to identify which body parts carry sentiment information in DGS—this interpretability focus is the technical differentiator from standard deep learning approaches that would likely achieve slightly higher accuracy but reveal nothing about the mechanism. The contribution is fundamentally a dataset artifact (DGS fairy tales with video and text sentiment labels) plus a methodology for cross-modal alignment: using LLM-annotated text as ground truth to train video-only models, effectively bootstrapping video understanding from text-based weak supervision.
How It Works
The pipeline has two distinct phases. Phase 1 (annotation): The authors take German fairy tale text, segment it into meaningful units, send each segment to four separate LLMs, and aggregate predictions via majority voting to assign a single sentiment label (negative/neutral/positive); they report 0.781 Krippendorff's alpha, indicating strong inter-annotator agreement even across multiple independent LLM annotators. Phase 2 (video feature extraction and training): For each text segment, they locate the corresponding video recording of the same text spoken in DGS, use MediaPipe (a real-time pose/hand/face detection library) to extract skeletal keypoints and facial landmarks frame-by-frame, aggregate these temporal features into fixed-dimension vector representations (likely using statistics like mean, std, or temporal pooling), and feed these motion vectors into XGBoost with the LLM-assigned labels as training targets. The XGBoost model outputs a three-class sentiment prediction and provides feature importance scores, revealing which body parts (eyebrows, mouth, hips, elbows) most strongly correlate with sentiment in DGS.
Production Impact
For a team building sign language understanding systems, this paper provides a concrete recipe: use LLM-annotated text as a scalable weak supervision source to label video data without manual annotation effort, then train interpretable models rather than opaque deep networks—this trades maybe 5-10% accuracy for explainability that regulators and domain experts (deaf community stakeholders) will trust. The 0.631 balanced accuracy is below production readiness for critical applications but is reasonable for assist systems (e.g., helping DGS learners, auto-captioning DGS content). Operationally, you'd run MediaPipe inference on video (fast, ~30 ms per frame on GPU), vectorize motion features once at indexing time, and use XGBoost for real-time prediction (<1 ms latency per inference); total compute is dominated by video processing, not sentiment prediction. The main integration complexity is ensuring proper video-text alignment—off-by-one segmentation errors will silently corrupt training labels, so robust video-to-text sync is critical infrastructure. You'd likely augment this with domain-specific tuning: DGS has regional variants, and signers have individual style differences, so deployed systems should either fine-tune per-region or accept that accuracy will degrade on out-of-distribution signers.
Limitations and When Not to Use This
The 0.631 balanced accuracy is frankly weak and limits real-world deployment to low-stakes scenarios like content discovery rather than accessibility-critical systems. The paper assumes strong video-to-text alignment and clear segmentation boundaries, but real-world DGS video is messy: overlapping signs, regional variation, signers with different body morphology all create distribution shift that isn't addressed. MediaPipe's pose estimation has documented failure modes on extreme poses and occlusions, and the paper doesn't report how often MediaPipe fails or how gracefully the system degrades when keypoints are missing—this is a production-critical unknown. The dataset is limited to fairy tales, a highly structured genre; generalization to everyday conversation, technical signing, or emotional DGS (e.g., storytelling with exaggerated affect) is completely untested. Finally, the paper doesn't address the elephant in the room: sentiment is arguably a Western linguistic construct, and whether it maps cleanly onto DGS grammar and expression is philosophically unclear—a domain expert in deaf linguistics should validate whether the three-class model even makes sense before deploying such a system.
Research Context
This work sits at the intersection of three research threads: (1) sentiment analysis in NLP, where using ensemble LLMs for weak annotation is increasingly common but typically applied to text-only tasks; (2) sign language NLP, a small but growing field that has historically focused on recognition/translation rather than semantic understanding; and (3) multimodal learning, where the core challenge is aligning video and text in meaningful ways. The paper builds on prior work using MediaPipe for sign language analysis (Holistic pose extraction) and ensemble methods for annotation, but applies them to a new domain and modality pairing. It opens up future directions in cross-lingual sentiment analysis (does the same sentiment manifest similarly in different sign languages?), temporal modeling of sentiment (is sentiment a frame-level property or does it evolve over signing sequences?), and domain transfer (can models trained on DGS transfer to American Sign Language or British Sign Language?). The dataset itself is the most likely to have lasting impact—researchers in accessibility tech and computational sociolinguistics may use it long after the XGBoost model is superseded.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::