AcademiClaw: When Students Set Challenges for AI Agents
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-04 with 13 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Junjie Yu et al. |
| Year | 2026 |
| HF Upvotes | 13 |
| arXiv | 2605.02661 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw.
Engineering Breakdown
Plain English
AcademiClaw introduces a 80-task benchmark for evaluating AI agents on university-level academic work—homework, research projects, competitions—submitted by 230 students and curated through expert review. The tasks span 25+ domains with hard requirements like 16 tasks needing GPU/CUDA execution, designed to stress-test current AI agents on problems students found they couldn't solve, filling a gap where existing OpenClaw benchmarks only tested assistant-level capabilities.
Key Engineering Insight
Real academic workflows require AI agents to handle multi-step, long-horizon problems that often demand specific computational resources (GPUs, specialized libraries) and domain expertise—not just instruction-following. The bilingual nature and diversity across olympiad math to systems debugging reveals that a single agent architecture likely won't solve the full range of production academic tasks without significant routing or specialization logic.
Why It Matters for Engineers
Production AI systems claiming academic capability need objective measurement against real student workflows, not synthetic tasks. This benchmark exposes failure modes in long-horizon reasoning, resource constraints, and domain-specific knowledge that matter when deploying agents into actual university environments or research settings where cost and reliability are measurable.
Research Context
Prior OpenClaw work evaluated AI agents on basic assistant tasks, leaving a blind spot around academic-complexity workflows. AcademiClaw closes that gap by grounding benchmark design in real, student-submitted failure cases rather than synthetic task design. This enables researchers to move beyond checking 'can the agent follow instructions' to 'can it actually tackle problems humans need solved,' raising the evaluation bar for production-readiness.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::