DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain
| Authors | Song Jin et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.10425 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent advancements in Vision-Language Models (VLMs) have revolutionized general visual understanding. However, their application in the food domain remains constrained by benchmarks that rely on coarse-grained categories, single-view imagery, and inaccurate metadata. To bridge this gap, we introduce DiningBench, a hierarchical, multi-view benchmark designed to evaluate VLMs across three levels of cognitive complexity: Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering. Unlike previous datasets, DiningBench comprises 3,021 distinct dishes with an average of 5.27 images per entry, incorporating fine-grained "hard" negatives from identical menus and rigorous, verification-based nutritional data. We conduct an extensive evaluation of 29 state-of-the-art open-source and proprietary models. Our experiments reveal that while current VLMs excel at general reasoning, they struggle significantly with fine-grained visual discrimination and precise nutritional reasoning. Furthermore, we systematically investigate the impact of multi-view inputs and Chain-of-Thought reasoning, identifying five primary failure modes. DiningBench serves as a challenging testbed to drive the next generation of food-centric VLM research. All codes are released in https://github.com/meituan/DiningBench.
Engineering Breakdown
Plain English
This paper introduces DiningBench, a new benchmark dataset for evaluating Vision-Language Models on food understanding tasks. The dataset contains 3,021 distinct dishes with an average of 5.27 images per dish, along with verified nutritional data. Unlike existing food datasets that use coarse-grained categories and single images, DiningBench provides three levels of evaluation: Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering. The authors evaluated 29 state-of-the-art VLMs against this benchmark to establish baseline performance and identify gaps in current model capabilities for food domain understanding.
Core Technical Contribution
The core contribution is a hierarchical, multi-view food benchmark that addresses specific limitations in existing VLM evaluation datasets. Traditional food benchmarks rely on coarse-grained category labels, single-view images, and unreliable metadata, which fail to capture the nuanced visual and nutritional understanding required for real-world applications. DiningBench introduces three key innovations: (1) fine-grained classification with hard negatives sourced from identical menus to force models to distinguish between visually similar dishes, (2) rigorous verification-based nutritional annotations rather than automatically scraped data, and (3) a multi-view paradigm with 5.27 images per dish enabling models to learn from multiple perspectives. This hierarchical structure progresses from visual recognition to semantic understanding to reasoning, creating a more comprehensive evaluation framework than prior single-task food datasets.
How It Works
The benchmark operates on a three-level cognitive hierarchy. At the base level, Fine-Grained Classification requires models to distinguish between visually similar dishes from the same menu, moving beyond coarse categories like 'salad' or 'pasta' to specific preparations and presentations. The second level, Nutrition Estimation, takes an image and produces quantitative predictions for calories, proteins, fats, carbohydrates, and other nutritional metrics, then compares against verified reference values. The third level, Visual Question Answering, presents an image with natural language questions requiring the model to reason about ingredients, preparation methods, dietary suitability, and other contextual information. For each of the 3,021 dishes, the benchmark provides multiple (average 5.27) high-quality images captured from different angles and lighting conditions, simulating realistic user inputs from phone cameras. The evaluation methodology measures classification accuracy, regression error for nutrition predictions, and answer correctness for VQA, with separate metrics for each complexity level to identify which cognitive tasks are challenging for current models.
Production Impact
For engineers building food-tech applications (delivery platforms, nutrition tracking, restaurant recommendation systems), this benchmark clarifies which VLM capabilities exist and where gaps remain. Before deployment, teams can run their vision pipeline against DiningBench's three task levels to understand whether off-the-shelf models like GPT-4V or Gemini can handle fine-grained dish recognition, and critically, whether nutritional estimation is reliable enough to claim dietary compliance. The multi-view requirement (5+ images per item) indicates that production systems may need to either acquire multiple training views or implement view-invariant architectures, increasing annotation and storage costs. The verified nutritional data component suggests that for any nutrition-critical application, synthetic web-scraped labels are insufficient—you'll need manual verification, a significant operational cost. The benchmark effectively enables teams to make data-driven decisions about whether to fine-tune models on domain data, use multi-modal retrieval augmented generation (RAG) with external nutrition databases, or wait for larger models to close capability gaps.
Limitations and When Not to Use This
The benchmark is limited to dish-level understanding and doesn't address the harder problem of ingredient-level detection or food quantity estimation from single images, which is critical for accurate calorie counting in real applications. The 3,021 dishes represent a curated, high-quality dataset that may not reflect the distribution of real user photos submitted to production systems, which often include poor lighting, partial views, mixed foods, and non-food context. The nutritional verification was likely conducted in controlled environments with specific preparation methods, so predictions on real-world, user-prepared variations of the same dish may exhibit significant drift. The paper assumes that fine-grained classification, nutrition estimation, and VQA are independent tasks, but in production these are often intertwined (e.g., you need to identify the dish to look up its nutrition), and the benchmark doesn't test this end-to-end composition. Additionally, the benchmark doesn't evaluate cross-cultural or regional cuisine diversity comprehensively, which limits its applicability to global food delivery and nutrition applications.
Research Context
This work builds on the broader trend of specialized benchmarks for Vision-Language Models, following examples like MMBench for general vision-language understanding and domain-specific evaluations in medical imaging and document analysis. DiningBench advances food understanding research beyond datasets like Food-101 and UEC Food-256, which use single images and coarse labels, toward multi-view, fine-grained evaluation. The hierarchical task design reflects growing recognition that VLM evaluation needs to progress from simple classification to semantic estimation to reasoning, mirroring how humans understand visual content. This benchmark will likely enable future work on domain adaptation techniques for VLMs, few-shot learning for new cuisines, and multi-modal fusion methods that combine vision with structured nutritional databases, establishing food domain understanding as a key testbed for VLM capabilities.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::