MMSkills: Towards Multimodal Skills for General Visual Agents
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 44 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Kangning Zhang et al. |
| Year | 2026 |
| HF Upvotes | 44 |
| arXiv | 2605.13527 |
| Download | |
| Code | https://github.com/DeepExperience/MMSkills |
Abstract
Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.
Engineering Breakdown
Plain English
This paper tackles how to build reusable skill packages for visual agents that go beyond text prompts or code—they need to understand visual state, recognize progress/failure, and decide next steps. The authors formalize 'multimodal procedural knowledge' and solve three practical problems: what should be in a skill package, how to extract skills from public interaction data, and how agents can use visual evidence efficiently at inference without blowing up image context.
Key Engineering Insight
Visual agent skills must be inherently multimodal packages containing visual state recognition, progress indicators, and decision logic—not just text instructions or code. This means treating skills as composable units that bundle vision understanding with procedural knowledge, which is fundamentally different from how most prompt-based agents work today.
Why It Matters for Engineers
Production visual agents (robotics, web automation, game-playing AI) fail at generalization because they can't reuse knowledge across tasks. Most systems either hard-code behaviors or use generic prompts that don't capture visual reasoning. This work directly addresses scaling agent capabilities through skill reuse without requiring massive prompt engineering or retraining for every new task.
Research Context
Prior work treated skills as text instructions or executable routines, which works for language but breaks down for agents that need visual perception. This paper advances the field by formalizing what visual procedural knowledge should look like and showing how to extract and compose these skills from public interaction data, enabling the next generation of agents that can rapidly adapt to new environments.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::