Encoder-Free Human Motion Understanding via Structured Motion Descriptions
| Authors | Yao Zhang et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.21668 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The world knowledge and reasoning capabilities of text-based large language models (LLMs) are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM's embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose Structured Motion Description (SMD), a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7% on BABEL-QA, 90.1% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. Code, data, and pretrained LoRA adapters are available at https://yaozhang182.github.io/motion-smd/.
Engineering Breakdown
Plain English
This paper proposes Structured Motion Description (SMD), a deterministic rule-based method that converts human motion sequences into natural language descriptions without needing dedicated motion encoders. Instead of learning cross-modal alignments through neural encoders, SMD leverages the reasoning capabilities of large language models by describing motion as structured text based on biomechanical principles—joint angles, body-part kinematics, and movement dynamics. The key insight is that text-based LLMs already possess broad world knowledge and reasoning abilities, so the bottleneck isn't model capability but rather how motion data is represented; by converting motion into precise natural language descriptions inspired by how biomechanists analyze movement, the authors enable LLMs to directly tackle motion understanding tasks like question answering and captioning without cross-modal projection layers.
Core Technical Contribution
The core novelty is replacing learned motion encoders with a deterministic rule-based system that translates motion sequences into structured natural language descriptions. Rather than training neural networks to align motion features with embedding spaces, SMD uses biomechanical principles—joint positions, angles, and kinematic chains—to generate descriptions that LLMs can naturally process. This eliminates the need for task-specific motion encoders and avoids the representation bottleneck that constrained prior LLM-based motion methods; the insight is that motion can be encoded as precise structured text that LLMs already understand through their pretraining on diverse language, making it a fundamentally different paradigm from encoder-based approaches.
How It Works
SMD takes as input raw joint position sequences (typically 3D coordinates from motion capture or pose estimation) and applies a rule-based conversion pipeline inspired by biomechanical analysis. The system extracts meaningful features from joint positions—computing angles between connected joints, tracking body-part positions relative to a root joint, and measuring movement velocities and accelerations. These numerical features are then converted into structured natural language descriptions using deterministic rules (e.g., 'the right knee angle is 45 degrees' or 'the left arm is moving upward at moderate speed'). The generated descriptions are formatted in a consistent, structured manner that preserves the temporal sequence of motion events. These descriptions are then fed directly into an LLM's input, which can reason about the motion using its pretrained knowledge without requiring any motion-specific encoders or embedding space alignment, making the entire pipeline encoder-free.
Production Impact
This approach offers significant practical advantages for production systems: it eliminates the need to train motion-specific encoders, reducing training complexity and data requirements substantially. Integration becomes simpler—you can use off-the-shelf LLMs without architectural modifications for motion tasks, just preprocessing motion data into structured descriptions beforehand. The computational cost during inference is lower since there are no encoder forward passes required; motion preprocessing is fast and rule-based. However, the trade-off is that the quality and expressiveness of motion descriptions depend entirely on the rule set, which requires careful design by domain experts familiar with biomechanics. For production pipelines, this means you'd precompute motion descriptions as text files, store them, and feed them to your LLM alongside question prompts—straightforward but requiring validation that your rule set captures all motion details relevant to your downstream tasks (QA, captioning, etc.).
Limitations and When Not to Use This
SMD is constrained by the expressiveness of its rule set—if the biomechanical features you encode don't capture motion nuances relevant to your task, the LLM can't infer missing information from raw motion data. The approach assumes that motion can be accurately described by joint angles and kinematic features, which may fail for subtle gestures, dynamic balance shifts, or social cues embedded in motion that aren't purely biomechanical. The deterministic nature means there's no learned adaptation to task-specific motion patterns; a general-purpose rule set must work across diverse motion understanding tasks, potentially suboptimally for any specific one. The paper doesn't address how to handle noisy pose estimation data (common in real-world scenarios) or motion at different scales and body sizes, and it's unclear whether the approach generalizes to cross-domain scenarios (e.g., describing animal motion or non-humanoid movement). Follow-up work would need to validate the rule set design empirically and explore how to make it adaptive or learnable while maintaining the encoder-free property.
Research Context
This paper builds on the broader trend of leveraging LLMs' reasoning and world knowledge for multimodal tasks, challenging the encoder-based paradigm that dominated earlier multimodal LLM work (e.g., CLIP, BLIP). It's inspired by classical biomechanics literature, which has long used joint angles and kinematic descriptions as a precise language for analyzing human movement, bringing that domain knowledge into the LLM era. The work advances motion understanding research (motion QA, motion captioning benchmarks) by showing that text-based LLMs can be competitive or superior without motion-specific encoders, suggesting that representation bottlenecks in prior work may have been architectural rather than fundamental. This opens a new research direction: exploring rule-based structured descriptions for other modalities (pose, gaze, hand gestures) and investigating how to jointly optimize description rules and LLM reasoning rather than treating them as completely decoupled.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::