QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
| Authors | Ali Slim et al. |
| Year | 2026 |
| HF Upvotes | 123 |
| arXiv | 2604.08570 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
Engineering Breakdown
Plain English
This paper introduces QuanBench+, a unified benchmark for evaluating how well large language models can generate quantum computing code across three major frameworks—Qiskit, PennyLane, and Cirq—with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. The authors measure model performance using executable functional tests and report both Pass@1 and Pass@5 metrics, plus a KL-divergence-based acceptance criterion for probabilistic quantum outputs. Their evaluation reveals significant performance gaps across frameworks: Qiskit reaches 59.5% Pass@1, Cirq achieves 54.8%, and PennyLane lags at 42.9%, suggesting models struggle differently depending on framework-specific syntax and conventions. They further study how performance improves when models can revise their code after receiving runtime error feedback or test failure information.
Core Technical Contribution
The core novelty is creating the first cross-framework unified benchmark for quantum code generation that isolates quantum reasoning ability from framework-specific knowledge. Rather than evaluating models only on single quantum frameworks—which conflates quantum understanding with API familiarity—QuanBench+ uses 42 semantically aligned tasks across three frameworks, allowing researchers to measure genuine quantum reasoning capabilities. The authors introduce KL-divergence-based acceptance for quantum probabilistic outputs, recognizing that quantum simulation results are inherently stochastic and requiring evaluation metrics beyond simple string matching. The feedback-based repair evaluation is a second key contribution, measuring how much improvement comes from automated error recovery, which is practically important for real code generation systems.
How It Works
The benchmark construction begins with 42 quantum computing tasks carefully aligned across Qiskit, PennyLane, and Cirq implementations, ensuring identical problem specifications but framework-specific code variants. For each task, models generate code solutions that are then executed against a test harness that verifies correctness via simulation and measurement statistics. The evaluation pipeline checks Pass@1 (does the first generated solution pass all tests) and Pass@5 (does at least one of five attempts pass), with passing defined as matching expected quantum output distributions within a KL-divergence threshold rather than exact string equivalence. The feedback-based repair mechanism feeds runtime errors and test failures back to the model, allowing it to see execution logs and retry, simulating the iterative code refinement that happens in practice. Measurements are recorded for each model-framework pair, producing a matrix that reveals both model quality and framework-specific difficulty, since the same model performs differently on identical quantum problems dressed in different syntactic frameworks.
Production Impact
For teams building quantum application generators or AI-assisted quantum development tools, this benchmark provides a standardized evaluation suite that prevents overfitting to a single framework and catches model weaknesses on real quantum problems. The KL-divergence metric is directly applicable to production systems: quantum code inherently produces probabilistic outputs, so accepting solutions by statistical equivalence rather than deterministic equality mirrors how actual quantum developers validate results. The feedback-based repair evaluation reveals that adding error recovery can significantly boost effective accuracy; production systems could implement this by capturing compilation errors, runtime failures, and test mismatches, then prompting the model to fix them—similar to existing code-refinement patterns in software engineering. However, integrating three quantum simulators (Qiskit, Cirq, PennyLane) adds infrastructure overhead; teams must decide whether framework-agnostic training justifies maintaining multiple testing backends. The 42-task dataset is relatively modest compared to large code benchmarks, so production models may need additional task-specific fine-tuning beyond what QuanBench+ can validate.
Limitations and When Not to Use This
The benchmark is limited to simulatable quantum circuits of modest size; real quantum hardware has connectivity constraints and noise models that this evaluation does not capture, so high Pass@1 scores on QuanBench+ do not guarantee practical utility on real quantum devices. The 42 aligned tasks, while carefully designed, may not cover emerging quantum algorithms or hybrid classical-quantum patterns seen in production quantum software, potentially missing important failure modes. The paper assumes models are LLMs fine-tuned or prompted for code generation but does not deeply explore how quantum-specific pre-training or architecture modifications might improve results, leaving open the question of what a quantum-specialized model might achieve. Additionally, evaluating only three frameworks excludes other important quantum platforms (Ocean/D-Wave, Silq, Q#), so results may not generalize to other quantum software ecosystems, and the feedback-based repair assumes models can improve via error messages without studying what types of errors are recoverable versus what require deeper semantic understanding.
Research Context
This work builds on the broader trend of benchmarking code generation models (HumanEval, MBPP, CodeXGLUE) but applies the methodology to the specialized domain of quantum computing, where framework heterogeneity is a major practical problem. It addresses a real gap: prior quantum benchmarks typically evaluate only Qiskit or PennyLane in isolation, making it impossible to separate quantum reasoning from framework syntax knowledge—a problem analogous to multilingual code benchmarks that discovered models overfit to one language. The unified benchmark approach echoes recent work on language-agnostic code evaluation, but quantum computing adds novel demands: the need for stochastic correctness metrics (KL-divergence), domain-specific error types, and validation that models understand quantum primitives like superposition and entanglement. The paper opens research directions around quantum-specialized model training, understanding how transfer learning across quantum frameworks works, and studying whether architectural innovations for quantum code (e.g., framework-agnostic embeddings) could improve cross-platform generalization.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::