InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
| Authors | Qiyao Wang et al. |
| Year | 2026 |
| HF Upvotes | 9 |
| arXiv | 2604.27419 |
| Download | |
| Code | https://github.com/AIforIP/InteractWeb-Bench |
Abstract
With the advancement of multimodal large language models (MLLMs) and coding agents, the website development has shifted from manual programming to agent-based project-level code synthesis. Existing benchmarks rely on idealized assumptions, especially for well-structured, information-rich inputs and static execution settings. In contrast, real-world development is constrained by a critical bottleneck: the semantic misalignment between ambiguous, low-quality instructions from non-expert users and model understanding, which results in a failure mode that we term blind execution. To address this gap, we introduce InteractWeb-Bench, the first multimodal interactive benchmark for website generation under non-expert low-code user conditions. InteractWeb-Bench introduces four types of user agents and persona-driven instruction perturbations to systematically simulate diverse user behaviors, including ambiguity, redundancy, and contradiction, grounded in requirement engineering defect taxonomies. We develop an interactive execution environment for agents, featuring a unified action space comprising Clarify, Implement, Verify, and Submit, enabling iterative intent refinement, code synthesis, and visual feedback-based validation. Extensive experiments and analysis reveal that frontier MLLM-based agents remain trapped in blind execution, exposing limitations in intent recognition and adaptive interaction.
Engineering Breakdown
Plain English
This paper introduces InteractWeb-Bench, a new benchmark for evaluating how well multimodal large language models (MLLMs) and coding agents can generate websites when given ambiguous, low-quality instructions from non-expert users—a scenario that existing benchmarks don't adequately test. The authors identify a critical failure mode they call 'blind execution,' where models misalign with what users actually want because the user instructions are vague or poorly structured, leading to incorrect implementations. InteractWeb-Bench addresses this gap by introducing four types of simulated user agents and interactive evaluation conditions that reflect real-world website development constraints. This is the first benchmark to systematically measure how well AI agents handle the semantic gap between imprecise user requirements and code generation at project scale.
Core Technical Contribution
The core contribution is identifying and formalizing the 'blind execution' problem—the mismatch between ambiguous user instructions and model interpretation—which existing benchmarks miss because they assume well-structured, information-rich inputs and static execution environments. The authors introduce InteractWeb-Bench as the first interactive, multimodal benchmark that simulates realistic non-expert user behavior through four agent types, moving beyond idealized evaluation settings. The benchmark combines interactive feedback loops and diverse user agent behaviors to measure how robustly MLLMs and code agents can clarify requirements, iterate on implementations, and converge to correct solutions. This shifts the evaluation paradigm from static code-to-specification matching to dynamic user-agent interaction, which is fundamentally closer to how real development workflows operate.
How It Works
InteractWeb-Bench operates as an interactive evaluation framework where a user agent submits an initial imprecise or ambiguous website requirement to an MLLM or coding agent, and the system must iteratively clarify, generate code, and refine based on feedback rather than producing output in a single pass. The benchmark includes four types of user agents (likely varying in expertise level, communication style, or specification quality) that simulate realistic conditions where non-expert users provide low-code, natural-language-like instructions with semantic gaps, missing details, or implicit assumptions. The MLLM/agent receives multimodal inputs (text descriptions, images, wireframes, or sketches) and must decide whether to request clarification, propose a solution, or iterate on previous attempts. Evaluation metrics measure not just final code correctness but also the agent's ability to detect misalignment, ask clarifying questions, and successfully converge to a correct website implementation over multiple interaction rounds. The benchmark likely includes reference implementations and success criteria that define when a generated website correctly interprets the user's intent, even if the original instruction was vague.
Production Impact
For teams building AI-assisted website builders or low-code platforms, this benchmark reveals that naive end-to-end code generation fails in realistic conditions where users cannot articulate requirements precisely—meaning production systems must implement interactive clarification loops and uncertainty detection rather than one-shot generation. Adopting the insights from this work means investing in agent architectures that can recognize ambiguous requests, ask targeted follow-up questions, and maintain context across multiple interaction rounds, significantly increasing system complexity compared to static code generation pipelines. The 'blind execution' failure mode directly affects user satisfaction and system reliability; in production, you'd need to add safeguards like asking users to confirm generated solutions, implementing rollback mechanisms for failed deployments, and providing explicit feedback mechanisms to help users refine specifications. This changes the compute and latency profile: instead of a single inference pass, you're now doing multiple shorter inference rounds with human feedback loops, which reduces peak latency per step but increases total time-to-completion and requires persistent session state management. Teams deploying this approach should budget for higher infrastructure costs due to iterative interactions, more sophisticated model architectures capable of uncertainty reasoning, and substantial user experience design to make the clarification process intuitive rather than frustrating.
Limitations and When Not to Use This
The paper focuses on website generation specifically, so insights may not generalize to other code synthesis tasks with different semantic complexity profiles, different user expertise distributions, or domains where specifications are inherently more deterministic. InteractWeb-Bench assumes that the four user agent types adequately represent real-world user behavior, but actual non-expert users may exhibit diverse communication patterns, learning curves, and intention-specification gaps that aren't fully captured by a discrete set of simulated agents. The benchmark doesn't address computational efficiency or scalability concerns—systems that pass InteractWeb-Bench may still be too expensive or slow for real production deployment, particularly if interactive loops require expensive model calls or long inference times. The paper doesn't solve the fundamental problem of measuring ground truth for 'correct' website implementations when user intent is intentionally ambiguous; evaluation relies on predefined reference implementations which may not capture all valid solutions or may encode arbitrary design choices. Follow-up work is needed to understand whether agents can be trained or fine-tuned specifically for clarification-based interaction (rather than just evaluated on it), and to measure performance across different model scales, architectures, and training approaches.
Research Context
This work builds on the growing research trajectory of project-level code synthesis and agent-based development, extending beyond earlier benchmarks like HumanEval and MBPP that focus on isolated functions rather than full website projects with complex dependencies and user interaction. It addresses a recognized gap in evaluation methodology: existing MLLM and code agent benchmarks (particularly those for webpage generation) typically assume perfect specification clarity and don't measure robustness to realistic user input quality, which has become increasingly important as these systems move toward user-facing applications. The paper opens a new research direction around interactive code generation and human-AI collaboration, where the benchmark serves as a testing ground for techniques like in-context clarification, uncertainty quantification, and multi-turn refinement loops. This work likely influences future benchmark design across the broader code generation community, establishing interactive, user-aligned evaluation as a standard rather than an afterthought, and emphasizes that agent capabilities must be measured not just on perfect-information tasks but on messy, real-world specification scenarios.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::