GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
| Authors | Shufan Jiang et al. |
| Year | 2026 |
| HF Upvotes | 44 |
| arXiv | 2604.02648 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.
Engineering Breakdown
Plain English
This paper introduces GBQA, a benchmark for evaluating whether large language models can autonomously detect bugs in software, using game development as the test domain. The benchmark contains 30 games with 124 human-verified bugs across three difficulty levels, constructed using a multi-agent system that both develops games and injects bugs at scale. The authors provide a baseline interactive agent equipped with a multi-round ReAct loop and memory mechanisms to perform bug detection. This addresses a critical gap: while LLMs have been extensively evaluated on code generation, their ability to discover bugs in complex dynamic environments remains largely unexplored, making this the first systematic benchmark for LLM-based quality assurance.
Core Technical Contribution
The core novelty is the scalable, human-verified benchmark construction methodology using multi-agent systems to both generate games and inject realistic bugs, enabling systematic evaluation of LLM bug-detection capabilities. Unlike prior work that focuses on code generation or static analysis, GBQA targets the harder problem of dynamic runtime bug discovery where LLMs must reason about program behavior across interactive environments. The baseline agent combines ReAct (Reasoning + Acting) with memory mechanisms specifically designed for iterative exploration and hypothesis testing in interactive debugging scenarios. The three-tier difficulty stratification (not just binary pass/fail) allows fine-grained measurement of LLM performance across complexity ranges, providing a more nuanced evaluation signal than existing code quality benchmarks.
How It Works
The benchmark construction uses a multi-agent pipeline where one agent generates game code (using game development frameworks or templates), another agent systematically injects bugs at specific code locations with known semantics, and human experts verify both the game functionality and bug authenticity before inclusion. The baseline interactive agent operates in a multi-round loop: it receives an environment description, formulates a hypothesis about potential bugs, selects an action to test that hypothesis (e.g., trigger a game event, examine program state), observes the outcome, updates its internal memory with findings, and repeats. The ReAct component ensures the agent maintains transparent reasoning chains, making its debugging logic interpretable and verifiable. Memory is critical here—the agent must retain information about previously tested code paths and bugs to avoid redundant exploration and to build cumulative understanding of the codebase. The agent interacts with a game runtime that provides execution traces, error logs, and state snapshots, enabling it to correlate actions with observable failures.
Production Impact
Adopting LLM-based automated bug detection could reduce manual QA labor by automating the initial exploration phase of dynamic testing, though it would likely complement rather than replace human QA engineers. In production, you'd integrate this as a pre-commit or CI/CD stage: the agent runs against staged code, explores common failure modes interactively, and flags potential issues for human verification before release. The trade-offs are significant: the ReAct loop is compute-intensive (multiple API calls per bug investigation), making per-commit cost non-trivial for large codebases; latency per bug investigation could be minutes (compared to seconds for static analysis), so this suits nightly CI runs rather than blocking gates; and the agent's effectiveness depends heavily on environment observability (you need good logging, state inspection APIs, and deterministic repro steps, which not all codebases provide). For game studios and interactive software companies, this is especially valuable because traditional static analysis misses runtime-environment-specific bugs (state corruption, race conditions in networked scenarios, frame-timing issues), which is precisely where this approach excels.
Limitations and When Not to Use This
The benchmark is constrained to game development, so generalization to other domains (web services, embedded systems, distributed software) remains unclear—bug manifestation patterns may differ fundamentally across domains. The human verification step, while ensuring quality, limits scalability: the paper covers 124 bugs across 30 games, which is small compared to real-world codebases with thousands of latent bugs; creating benchmarks for new domains would require substantial expert effort. The baseline agent assumes access to detailed runtime observability (execution traces, state inspection); many legacy systems lack this instrumentation, reducing applicability. Critically, the paper doesn't measure false positive rates or the cost of the agent wasting time on non-bugs, which is essential for production adoption—a single misclassified bug could introduce erroneous fixes. The evaluation methodology appears incomplete in the abstract (the description cuts off), so it's unclear whether the paper addresses how to handle bugs the agent discovers but humans disagree on, or how performance scales with codebase size.
Research Context
This work advances the recent trend of using LLMs for software engineering tasks beyond generation (which has been the focus of Codex, GitHub Copilot, and related research) into the harder, less-studied territory of testing and QA. It builds on ReAct (Reasoning + Acting in LLMs) and prior benchmarking work in code understanding (HumanEval, MBPP) but applies these to a new task—bug discovery in dynamic environments rather than static code completion. The paper fills a gap in the evaluation landscape: while there are benchmarks for code generation quality (measuring correctness of generated code) and vulnerability detection (static security analysis), there's been little systematic evaluation of LLM-based testing and dynamic bug discovery. This opens a research direction toward autonomous software quality engineering, potentially connecting to broader work in AI-assisted development environments and the integration of LLM agents into CI/CD pipelines.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::