ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue
| Authors | Daoxuan Zhang et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2605.01371 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The rapid advancement of Multimodal Large Language Models (MLLMs) has empowered Unmanned Aerial Vehicle (UAV) with exceptional capabilities in spatial reasoning, semantic understanding, and complex decision-making, making them inherently suited for UAV Search and Rescue (SAR). However, existing UAV SAR research is dominated by traditional vision and path-planning methods and lacks a comprehensive and unified benchmark for embodied agents. To bridge this gap, we first propose the novel task of Embodied Search and Rescue (ESAR), which requires aerial agents to autonomously explore complex environments, identify rescue clues, and reason about victim locations to execute informed decision-making. Additionally, we present ESARBench, the first comprehensive benchmark designed to evaluate MLLM-driven UAV agents in highly realistic SAR scenarios. Leveraging Unreal Engine 5 and AirSim, we construct four high-fidelity, large-scale open environments mapped directly from real-world Geographic Information System (GIS) data to ensure photorealistic landscapes. To rigorously simulate actual rescue operations, our benchmark incorporates dynamic variables including weather conditions, time of day, and stochastic clue placement. Furthermore, we create a dataset of 600 tasks modeled after real-world rescue cases and propose a robust set of evaluation metrics. We evaluate diverse baselines, ranging from traditional heuristics to advanced ground and aerial MLLM-based ObjectNav agents. Experimental results highlight the challenges in ESAR, revealing critical bottlenecks in spatial memory, aerial adaptation, and the trade-off between search efficiency and flight safety. We hope ESARBench serves as a valuable resource to advance research on Embodied Search and Rescue domain. Source code and project page: https://4amgodvzx.github.io/ESAR.github.io.
Engineering Breakdown
Plain English
This paper introduces ESARBench, the first comprehensive benchmark for evaluating multimodal large language models (MLLMs) deployed on autonomous UAVs for search and rescue missions. The authors define a new task called Embodied Search and Rescue (ESAR) where aerial agents must explore complex environments, identify rescue clues from visual and spatial data, and reason about victim locations to make informed decisions. The benchmark addresses a critical gap in UAV research: while traditional vision and path-planning methods dominate SAR applications, there's no unified evaluation framework for MLLM-driven embodied agents that combines perception, reasoning, and autonomous decision-making. This work provides both the task formulation and evaluation infrastructure needed to measure how well modern multimodal models can handle real-world SAR scenarios.
Core Technical Contribution
The paper's core novelty is formulating Embodied Search and Rescue (ESAR) as a structured benchmark task that unifies three traditionally separate research areas: embodied AI, multimodal reasoning, and UAV autonomy. Unlike prior UAV work that relies on hand-crafted vision pipelines and separate planning modules, this approach leverages MLLM capabilities to handle spatial reasoning, semantic understanding, and decision-making in a single agent. ESARBench is the first benchmark to systematically evaluate MLLMs in this embodied SAR context, providing standardized environments, evaluation metrics, and baseline implementations. The contribution is primarily in problem formulation and benchmark design rather than novel algorithms—it establishes what metrics matter for real SAR scenarios and how to measure MLLM performance on spatial reasoning and victim localization tasks.
How It Works
The ESAR task operates in a structured pipeline: a UAV equipped with an MLLM receives visual observations from its onboard camera as it autonomously navigates a simulated or real environment. At each step, the MLLM processes the current visual frame and spatial context (position, orientation, altitude) to identify rescue clues—injured people, distress signals, environmental hazards—and reason about victim locations based on accumulated observations. The agent maintains a spatial memory or map of explored areas and clues, using the MLLM's reasoning capabilities to infer where victims are likely located and whether immediate rescue is feasible. The benchmark provides standardized environments (likely procedurally generated or pre-recorded UAV trajectories), success criteria (correctly identifying victim locations, optimizing search time), and baseline implementations using state-of-the-art MLLMs like GPT-4V or similar models. Evaluation metrics track both perception accuracy (clue detection) and decision quality (victim location predictions), with ground truth victim positions annotated in the benchmark dataset.
Production Impact
For engineers building real SAR systems, this benchmark establishes quantifiable performance targets and evaluation protocols for MLLM-based UAV agents, moving beyond ad-hoc testing toward reproducible benchmarking. Production adoption would require integrating an MLLM inference engine (likely cloud-based due to model size) with UAV navigation stacks, which introduces latency trade-offs—real-time SAR demands sub-second response times while large model inference can add hundreds of milliseconds. The benchmark helps teams identify which MLLM families and sizes are viable for embedded UAV deployment, informing decisions about model quantization, distillation, or edge inference optimization. On the integration side, practitioners would need to implement robust spatial mapping layers, camera calibration, and fallback planning for safety-critical scenarios where MLLM reasoning might fail or hallucinate victim locations. This work also provides a shared evaluation framework for comparing different MLLM architectures and prompting strategies, reducing duplicated engineering effort across research teams and companies building SAR solutions.
Limitations and When Not to Use This
The paper's abstract doesn't specify details about the synthetic vs. real-world data split in ESARBench, which is critical since MLLM performance often degrades significantly on real UAV footage compared to clean synthetic renders. The benchmark likely assumes ideal conditions—good lighting, clear victim visibility, controlled environment layouts—that may not reflect chaotic real disaster scenes with obscured casualties, weather interference, and dynamic obstacles. There's no discussion of failure modes specific to SAR: hallucinating false victims, missing obscured casualties, or misidentifying debris as people, which are especially dangerous in life-critical applications. The approach also doesn't address the compute and latency constraints of actual deployed UAVs; running inference on modern MLLMs requires substantial hardware, and the paper doesn't clarify whether edge deployment or cloud offloading is assumed, which dramatically affects practical viability. Additionally, the reliance on MLLM reasoning for safety-critical decisions raises liability and safety validation questions that benchmarking alone cannot resolve.
Research Context
This work builds on the recent wave of research demonstrating MLLM capabilities in spatial reasoning and embodied AI (e.g., LLaVA, GPT-4V) and extends them to an underexplored domain: autonomous UAV search and rescue. It advances prior UAV SAR research, which has traditionally relied on hand-crafted computer vision pipelines, SLAM for mapping, and separate classical planning algorithms—this paper proposes replacing those modular systems with unified MLLM reasoning. The benchmark format follows successful patterns from other embodied AI evaluations like Habitat 2.0 and EmbodiedQA, adapting that evaluation paradigm to aerial robotics and SAR-specific tasks. The contribution opens a new research direction: how to optimize MLLMs specifically for embodied spatial reasoning under resource constraints, potentially spawning follow-up work on model distillation, prompt engineering for SAR, and hybrid architectures that combine MLLMs with lightweight classical planning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::