BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
| Authors | Jiacheng Shen et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.16241 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Large language models have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language models handle specialized animal-related knowledge under a unified closed-book evaluation protocol. We introduce BAGEL, a benchmark for evaluating animal knowledge expertise in language models. BAGEL is constructed from diverse scientific and reference sources, including bioRxiv, Global Biotic Interactions, Xeno-canto, and Wikipedia, using a combination of curated examples and automatically generated closed-book question-answer pairs. The benchmark covers multiple aspects of animal knowledge, including taxonomy, morphology, habitat, behavior, vocalization, geographic distribution, and species interactions. By focusing on closed-book evaluation, BAGEL measures animal-related knowledge of models without external retrieval at inference time. BAGEL further supports fine-grained analysis across source domains, taxonomic groups, and knowledge categories, enabling a more precise characterization of model strengths and systematic failure modes. Our benchmark provides a new testbed for studying domain-specific knowledge generalization in language models and for improving their reliability in biodiversity-related applications.
Engineering Breakdown
Plain English
BAGEL is a new benchmark designed to test how well large language models answer specialized questions about animals across seven knowledge dimensions: taxonomy, morphology, habitat, behavior, vocalization, geographic distribution, and species interactions. The benchmark combines curated examples and automatically generated question-answer pairs from scientific sources like bioRxiv, Global Biotic Interactions, Xeno-canto, and Wikipedia, using a closed-book evaluation protocol where models must answer without external references. This addresses a gap in existing benchmarks, which focus on broad-domain knowledge but don't systematically evaluate specialized domain expertise. The work reveals how current LLMs perform on niche, scientifically rigorous animal knowledge that requires deep subject matter understanding.
Core Technical Contribution
The core contribution is the BAGEL benchmark itself—a systematic, multi-source evaluation framework that isolates animal knowledge expertise from general reasoning ability. Unlike prior benchmarks that mix broad knowledge with reasoning tasks, BAGEL uses closed-book evaluation to force models to demonstrate what they actually know about animals, not their ability to search or infer. The benchmark combines human curation with automatic question generation to scale across multiple knowledge dimensions while maintaining quality control. This represents the first comprehensive benchmark that systematically measures specialized biological domain knowledge across diverse animal taxa and behavioral/ecological phenomena.
How It Works
BAGEL's construction pipeline starts by aggregating diverse authoritative sources: bioRxiv provides recent scientific publications, Global Biotic Interactions supplies ecological relationship data, Xeno-canto offers bird vocalization information, and Wikipedia provides structured reference content. The team then creates two types of evaluation items: carefully curated questions from domain experts ensure high-quality coverage of key concepts, while automatically generated question-answer pairs scale the benchmark across more species and knowledge types. The closed-book protocol means models receive only the question text with no external retrieval augmentation—they must generate answers using only their parametric knowledge. The benchmark organizes questions across seven knowledge dimensions (taxonomy, morphology, habitat, behavior, vocalization, distribution, interactions), allowing analysis of which knowledge types LLMs handle well or poorly. Evaluation metrics compare model outputs against reference answers, likely using exact match and semantic similarity scores to handle valid answer variation.
Production Impact
For teams building domain-specific AI applications—such as wildlife monitoring systems, ecological research assistants, or veterinary diagnostic tools—BAGEL provides a concrete way to measure whether your base model actually knows the domain or just appears confident. You can run your models against BAGEL before deployment to identify knowledge gaps and decide whether to fine-tune, use retrieval augmentation, or invest in retraining. The closed-book evaluation is particularly valuable because it shows whether your model will fail gracefully when access to external references is unavailable (edge devices, offline scenarios, latency-critical applications). The multi-dimensional structure lets you profile exactly which knowledge types your model struggles with—for example, you might find your model knows taxonomy well but fails on vocalization or species interactions—informing targeted improvement strategies. However, building or extending this benchmark requires access to high-quality domain expert curation, and the evaluation overhead (manual annotation for quality assessment) means running comprehensive BAGEL evaluation regularly is expensive.
Limitations and When Not to Use This
BAGEL's closed-book evaluation assumes that parametric knowledge is the right metric, but in practice many production systems combine models with retrieval to handle both known and novel species—this benchmark doesn't evaluate that hybrid approach. The benchmark covers primarily well-studied animal groups (taxa with bioRxiv publications and Xeno-canto recordings), likely underrepresenting insects, deep-sea organisms, and other poorly-documented groups, which limits its applicability to full taxonomic diversity. The automatic question generation component introduces potential bias toward questions that are easy to generate programmatically rather than representative of real information needs in wildlife biology or ecology. The paper doesn't specify how it handles multi-modal animal knowledge (visual identification from photographs, audio classification of vocalizations), which are critical in real wildlife applications, so BAGEL appears to be text-only evaluation. Finally, this is a static benchmark frozen at a point in time, while animal knowledge in scientific literature continuously evolves, meaning benchmark questions may become outdated as new species are discovered or behavior patterns are better understood.
Research Context
BAGEL builds on a wave of recent domain-specific benchmarking work that challenges the assumption that large language models have uniform knowledge—prior work created specialized benchmarks for biomedical knowledge (PubMedQA), law (LegalBench), and mathematics (MATH), all finding significant performance gaps. The work responds to concerns in the LLM community that broad-domain benchmarks like MMLU don't adequately measure specialized expertise, and that models appear more knowledgeable than they actually are on niche topics. BAGEL's multi-source, multi-dimensional approach follows the design philosophy of recent diverse-source benchmarks but applies it specifically to the biological sciences and animal behavior, creating a systematic evaluation framework for evaluating models on structured domain knowledge. This research opens directions for similar benchmarks in other scientific domains (marine biology, entomology, botany) and raises questions about how to effectively teach LLMs specialized knowledge through fine-tuning, retrieval augmentation, or architecture modifications.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::