MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
| Authors | Tanmay Gupta et al. |
| Year | 2026 |
| HF Upvotes | 37 |
| arXiv | 2604.08516 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.
Engineering Breakdown
Plain English
This paper addresses the lack of open, reproducible web agents by introducing MolmoWeb, a family of fully open multimodal models trained to autonomously navigate and complete tasks on websites. The authors created MolmoWebMix, a large training dataset combining over 100K synthetic task trajectories from multiple generation pipelines with 30K+ human demonstrations and atomic web-skill annotations. The contribution is significant because existing state-of-the-art web agents rely on proprietary closed models and undisclosed training data, making scientific progress and community-driven development difficult. By releasing both the models and the diverse, well-annotated training data openly, the authors enable reproducible research and lower barriers to entry for the web agent research community.
Core Technical Contribution
The core innovation is the combination of two complementary technical contributions: (1) MolmoWebMix, a hybrid dataset that mixes synthetic trajectories from multiple complementary generation pipelines with high-quality human demonstrations, and (2) MolmoWeb, a family of open multimodal foundation models fine-tuned for web task execution. Unlike prior work that either uses proprietary models or limited training data, this approach scales task diversity by generating synthetic demonstrations from multiple sources while grounding them with real human examples, creating a more robust training signal. The paper demonstrates that you can build capable web agents without relying on proprietary closed models, fundamentally changing the economics of web agent development by making the entire pipeline open and reproducible.
How It Works
MolmoWeb agents operate as closed-loop systems that iteratively perceive the current webpage state, reason about task requirements, and execute actions. The architecture takes as input both the visual rendering of a webpage and textual context describing the user's objective, processes this through a multimodal foundation model (Molmo), and outputs a structured action specification (e.g., click coordinates, text input, navigation commands). The training process uses the MolmoWebMix dataset where each example contains a sequence of states, actions, and outcomes—synthetic examples are generated programmatically using task simulators and action compilers, while human demonstrations provide ground-truth expertise and edge-case coverage. During inference, the agent uses in-context learning or fine-tuning to condition on task descriptions, then generates actions autoregressively, with a reward or trajectory scoring mechanism (implicit in demonstrations) guiding the model toward successful completion. The key technical insight is that mixing synthetic and human data prevents the model from overfitting to any single generation pipeline's biases, improving generalization to unseen websites and task types.
Production Impact
For engineers building web automation systems, MolmoWeb removes the dependency on proprietary APIs (like those from OpenAI or Google) for web task execution, significantly reducing operational costs and vendor lock-in risk—you can now self-host and fine-tune open models on your own infrastructure. The open dataset (MolmoWebMix) provides a foundation for rapid iteration: teams can use it as a starting point, add domain-specific human demonstrations, and fine-tune a model in weeks rather than months, accelerating time-to-value for internal automation projects. However, production deployment requires careful consideration of hallucination rates, action failure recovery, and latency: web agents still make mistakes (clicking wrong buttons, misinterpreting dynamic content), so you'll need robust error handling, human-in-the-loop fallbacks, and monitoring systems. The compute footprint is substantial—multimodal models require GPU inference for each step of a task trajectory, so scaling to thousands of concurrent users demands significant infrastructure investment; caching and distillation techniques become critical for cost-effective deployment.
Limitations and When Not to Use This
The paper's scope is limited to web-GUI tasks and may not generalize to complex, multi-step reasoning tasks that require deep domain knowledge (e.g., financial analysis, medical diagnosis) where web scraping alone is insufficient. The synthetic trajectory generation relies on simulators and heuristics that may not cover rare edge cases or adversarial website layouts (dynamic JavaScript, infinite scroll, CAPTCHA), so real-world performance gaps are likely—the paper doesn't provide comprehensive failure analysis or metrics on out-of-distribution robustness. The 30K+ human demonstrations, while substantial, may have annotation quality variance and potential geographic or cultural biases in task selection, which aren't thoroughly characterized. Scalability to much larger datasets or real-time multi-agent coordination scenarios remains unexplored, and the paper lacks detailed analysis of how performance scales with model size or dataset size.
Research Context
This work builds on prior research in embodied AI agents and vision-language models (like those in robotics and autonomous driving) but adapts them to the web domain where the action space is discrete (clicks, text input) rather than continuous. It extends recent work on interactive learning and in-context learning in large language models (like Prompt2API and similar web automation papers) by emphasizing dataset diversity and open reproducibility—most prior web agents were proprietary or trained on limited, manually-collected data. The paper opens a new research direction around open-source foundation models for structured task execution, potentially influencing how the community builds agents for other domains (mobile apps, desktop environments). It also contributes to the broader movement toward open AI research, challenging the dominant paradigm of closed proprietary models and encouraging reproducible, community-driven development in AI applications.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::