OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction
| Authors | Junyoung Lee et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.28197 |
| Download | |
| Categories | cs.RO, cs.CV |
Abstract
Human-robot collaboration has been studied primarily in dyadic or sequential settings. However, real homes require multiadic collaboration, where multiple humans and robots share a workspace, acting concurrently on interleaved subtasks with tight spatial and temporal coupling. This regime remains underexplored because close-proximity interaction between humans, robots, and objects creates persistent occlusion and rapid state changes, making reliable real-time 3D tracking the central bottleneck. No existing platform provides the real-time, occlusion-robust, room-scale perception needed to make this regime experimentally tractable. We present OmniRobotHome, the first room-scale residential platform that unifies wide-area real-time 3D human and object perception with coordinated multi-robot actuation in a shared world frame. The system instruments a natural home environment with 48 hardware-synchronized RGB cameras for markerless, occlusion-robust tracking of multiple humans and objects, temporally aligned with two Franka arms that act on live scene state. Continuous capture within this consistent frame further supports long-horizon human behavior modeling from accumulated trajectories. The platform makes the multiadic collaboration regime experimentally tractable. We focus on two central problems: safety in shared human-robot environments and human-anticipatory robotic assistance, and show that real-time perception and accumulated behavior memory each yield measurable gains in both.
Engineering Breakdown
Plain English
OmniRobotHome is the first room-scale residential platform that solves the core perception bottleneck in multi-robot, multi-human collaborative tasks. The paper addresses a critical gap: existing systems handle dyadic (two-agent) or sequential collaboration, but real homes require multiple humans and robots working concurrently in tight spatial and temporal coupling with persistent occlusion. The platform unifies real-time 3D human and object perception with coordinated multi-robot control, enabling researchers to experimentally tackle multiadic (many-agent) collaboration at room scale—something previously infeasible due to the tracking reliability required in close-proximity, occluded scenarios.
Core Technical Contribution
The core novelty is the integration of wide-area, real-time, occlusion-robust 3D perception as the foundational layer for multi-robot coordination in shared human workspaces. Rather than treating perception as a module bolted onto existing robotics platforms, OmniRobotHome makes occlusion-handling and room-scale tracking the architectural centerpiece, recognizing that dense, concurrent human-robot-object interaction cannot be reliably managed without continuous, low-latency 3D state awareness. The contribution is not a single algorithm but a systems-level solution that unifies perception infrastructure with multi-robot orchestration—this is distinct from prior work that either solved perception in isolation, assumed limited occlusion, or operated in small workspaces where trajectory prediction could substitute for robust tracking.
How It Works
The system operates in a perception-action loop: wide-area 3D sensors (likely multi-camera rigs, depth sensors, or LiDAR arrays distributed across the room) continuously ingest visual and depth data. The perception pipeline must handle the core challenge—occlusion—by fusing multiple viewpoints and potentially using learned predictive models to infer occluded human and object positions even when line-of-sight is blocked. This real-time 3D state (human poses, object positions, robot poses) flows into a multi-robot coordination layer that must make decisions on a tight temporal budget: which robot acts on which subtask, when to pause to avoid collision, how to hand off objects or tasks between agents. The output is coordinated motion commands to multiple robots that respect both spatial constraints (no collisions) and task dependencies (sequential subtasks on interleaved objects), all updated at room-scale latency (likely <100ms to handle fast human movements).
Production Impact
For teams deploying multi-robot systems in human-shared spaces, this platform eliminates the need to build custom perception infrastructure from scratch—a massive engineering burden that has historically required specialized computer vision and sensor fusion teams. Production adoption would mean: (1) faster time-to-deployment for collaborative warehouse or fulfillment scenarios, since the perception and coordination stack is pre-integrated; (2) much higher collision safety, since occlusion-robust tracking eliminates 'blind spot' failures where robots move into unseen humans; (3) ability to scale from 2-robot demos to true multi-agent systems without rearchitecting perception. The trade-off is substantial: room-scale sensor coverage is expensive (multiple high-resolution cameras, depth sensors, or LiDAR units), and real-time occlusion handling adds compute overhead (GPU inference for pose prediction). Latency is critical—if perception loops exceed ~100ms, humans moving at natural speeds will outpace the system's awareness, breaking safety guarantees.
Limitations and When Not to Use This
The platform's scope is explicitly residential and room-scale; it does not address outdoor, mobile, or multi-room scenarios where maintaining continuous sensor coverage becomes infeasible. The paper assumes that dense sensor networks are deployable and maintained—a constraint that doesn't hold in many industrial or dynamic environments where sensors are occluded or unreliable. Occlusion robustness likely depends on learned models (e.g., pose predictors) trained on specific human activity distributions; these models may fail catastrophically on novel interactions or in cultural/biomechanical contexts outside the training set. Scalability to very large numbers of robots (>5–10) or very dense human activity is unvalidated; the coordination layer's computational complexity may grow superlinearly, and the assumption of tight temporal coupling may break down in chaotic, high-throughput settings.
Research Context
This work builds on decades of human-robot interaction research and multi-robot motion planning, but explicitly moves beyond the 'dyadic interaction' paradigm that has dominated HRI since early work by Yanco & Drury. It responds to the practical realization that real homes and shared workspaces are multiadic—multiple agents with asymmetric capabilities and goals must coordinate in real time. The contribution sits at the intersection of three research communities: embodied AI (perception for robotics), multi-agent planning (coordination), and human-robot safety (occlusion-aware tracking). By providing an experimental platform with unified perception and coordination, OmniRobotHome enables downstream research on task allocation, learning-in-the-loop adaptation, and human intent prediction in dense collaborative scenarios—all questions that were previously hard to even pose systematically.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::