Structured Distillation of Web Agent Capabilities Enables Generalization
| Authors | Xing Han Lù & Siva Reddy |
| Year | 2026 |
| HF Upvotes | 18 |
| arXiv | 2604.07776 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io
Engineering Breakdown
Plain English
This paper presents Agent-as-Annotators, a framework for training smaller language models to navigate and interact with websites by learning from synthetic trajectories generated by a larger teacher model. The authors use Gemini 3 Pro to generate 3,000 training trajectories across six web environments, then fine-tune a 9B-parameter student model on the 2,322 highest-quality trajectories using supervised learning. The resulting model achieves 41.5% accuracy on WebArena, outperforming closed-source models like Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%), and nearly doubling the previous best open-weight baseline (21.7%). This enables practical deployment of capable web agents without relying on expensive third-party APIs.
Core Technical Contribution
The core innovation is structuring synthetic trajectory generation for web agents by analogy to human annotation workflows, with modular LLM components replacing discrete human roles (Task Designer, Annotator, Supervisor). Rather than treating data generation as monolithic, the framework decomposes it into specialized roles that can each be implemented as LLM prompts, enabling systematic quality control and iterative refinement. This modular approach to synthetic data generation appears novel in the web agent domain and contrasts with prior work that either relied on human annotation or unstructured automated generation. The key insight is that treating the teacher model as performing structured annotation work—not just generating random trajectories—leads to significantly higher-quality training data and better downstream student performance.
How It Works
The framework operates as a structured pipeline with three modular LLM components: (1) Task Designer generates realistic web navigation tasks with diverse objectives, (2) Annotator executes these tasks using the teacher model (Gemini 3 Pro) to produce action sequences and trajectories, and (3) Supervisor evaluates and filters trajectories based on success criteria and quality metrics. For each task, the Annotator generates a full trajectory—a sequence of states, observations, and actions taken to navigate a website and complete the objective. Only trajectories that pass the Supervisor's filtering criteria (2,322 out of 3,000 initial generation, a ~77% pass rate) are retained for training. The student model, a 9B-parameter transformer, is then fine-tuned using pure supervised learning on these filtered trajectories, learning to predict the correct next action given the current state and task. The training uses these structured (state, action) pairs as a standard sequence-to-sequence learning objective.
Production Impact
For engineers deploying web agents in production, this work provides a concrete path to cost-effective local inference: instead of routing every request to an expensive API-based model like Claude or GPT-4, you can now fine-tune and deploy a 9B open-weight model that matches or exceeds their performance. This cuts per-inference cost by 10-100x depending on your baseline and enables full data privacy and latency control—critical for enterprise deployments. The modular annotation framework is also reusable: teams can apply the same Task Designer/Annotator/Supervisor structure to their own web domains and workflows, generating domain-specific training data without manual annotation effort. The trade-off is that you need compute resources for the initial trajectory generation phase (3,000 trajectories with Gemini 3 Pro), storage for training data, and GPU memory for fine-tuning a 9B model (likely requiring an A100 or equivalent). For companies processing high volumes of web tasks, the ROI is substantial; for low-volume use cases, sticking with API calls may remain more cost-effective.
Limitations and When Not to Use This
The approach assumes that synthetic trajectories generated by a larger teacher model transfer well to diverse web environments—but distribution shift between training domains (the six environments used) and new, unseen websites remains a concern not fully addressed. The filtering rate (77% of generated trajectories passing QA) suggests that the Supervisor's quality criteria are still imperfect, and filtering methods are not deeply analyzed; better filtering or trajectory refinement techniques could improve the pass rate and reduce wasted generation effort. The paper does not measure latency or throughput on web agent tasks, so real-time performance and interaction speed are unknown—web navigation may require fast action generation that 9B models cannot provide compared to larger models. Additionally, the framework is evaluated only on WebArena; generalization to other web agent benchmarks or to real-world production websites with novel UI patterns, JavaScript-heavy interactions, or CAPTCHA challenges is not demonstrated. The paper also does not explore how to handle tasks requiring reasoning across multiple pages, multi-step planning, or recovery from errors—capabilities that may be needed in production but are not explicitly evaluated.
Research Context
This work builds on a growing body of research in distillation for specialized tasks (e.g., distilling large models into smaller ones for deployment) and synthetic data generation for training. It advances the web agent domain, which has seen recent progress with models like Claude 3.5 Sonnet and approaches like CoAct, by showing that open-weight models can be made competitive with closed-source models through careful distillation. The structured annotation framework is influenced by human annotation pipelines used in datasets like WebArena and WebShop, translating those human workflows into LLM components—this analogy is conceptually clean and suggests future work in decomposing other complex tasks into modular LLM roles. The results also contribute to the broader trend of production-oriented ML engineering: as frontier models become expensive and API-dependent, the ability to distill and finetune smaller models for specific domains becomes critical infrastructure. This opens research questions around optimal filtering criteria for synthetic data, multi-task distillation for web agents, and how to systematically expand to new web domains without complete retraining.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::