SG-DOR: Learning Scene Graphs with Direction-Conditioned Occlusion Reasoning for Pepper Plants
| Authors | Rohit Menon et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2603.06512 |
| Download | |
| Categories | cs.RO, cs.CV |
Abstract
Robotic harvesting in dense crop canopies requires effective interventions that depend not only on geometry, but also on explicit, direction-conditioned relations identifying which organs obstruct a target fruit. We present SG-DOR (Scene Graphs with Direction-Conditioned Occlusion Reasoning), a relational framework that, given instance-segmented organ point clouds, infers a scene graph encoding physical attachments and direction-conditioned occlusion. We introduce an occlusion ranking task for retrieving and ranking candidate leaves for a target fruit and approach direction, and propose a direction-aware graph neural architecture with per-fruit leaf-set attention and union-level aggregation. Experiments on a multi-plant synthetic pepper dataset show improved occlusion prediction (F1=0.73, NDCG@3=0.85) and attachment inference (edge F1=0.83) over strong ablations, yielding a structured relational signal for downstream intervention planning.
Engineering Breakdown
Plain English
This paper addresses robotic harvesting in dense crop environments where picking fruit requires understanding not just the 3D geometry of plants, but which leaves and stems physically block access to target fruits from different approach directions. The authors propose SG-DOR, a framework that takes point cloud segmentations of plant organs and builds a scene graph representing physical attachments between parts, then predicts which leaves obstruct each fruit depending on the picking direction. On a synthetic pepper dataset, their direction-aware graph neural network achieves F1=0.73 for occlusion prediction, outperforming baselines by explicitly reasoning about direction-conditioned spatial relationships rather than treating occlusion as a static property.
Core Technical Contribution
The core novelty is introducing direction-conditioned occlusion reasoning to robotic manipulation—the insight that whether a leaf blocks a fruit depends critically on the approach vector, not just absolute geometry. Instead of building a static scene graph, SG-DOR conditions occlusion predictions on the approach direction and proposes an occlusion ranking task that retrieves and orders candidate obstructing leaves for a specific fruit-direction pair. The technical contribution is a direction-aware graph neural architecture with per-fruit leaf-set attention mechanisms and union-level aggregation, which allows the model to learn that different subsets of leaves matter depending on the picking angle. This is a departure from prior work that either ignores occlusion entirely or treats it as direction-agnostic.
How It Works
The pipeline starts with instance-segmented organ point clouds as input—each plant part (fruit, leaf, stem) is separately segmented. The system first constructs a static scene graph encoding physical attachment relationships (which leaf connects to which branch, which fruit to which stem). Then, for a target fruit and a given approach direction vector, the direction-aware graph neural network performs per-fruit leaf-set attention: it learns to weight which leaves in the plant's neighborhood are relevant obstructions for that specific approach. The model aggregates these attention weights at the union level to produce a ranking of leaves by occlusion probability. The occlusion ranking task trains the network to order candidate leaves such that the most-obstructing leaves for a given approach direction appear first in the ranking, enabling downstream planners to decide on trajectory adjustments or alternative picking strategies.
Production Impact
For a robotic harvesting system, this approach directly improves path planning and grasp success rates by predicting precisely which plant parts must be moved or cut before picking a target fruit from a particular angle. Instead of a gripper attempting a blind pick and failing when a leaf wraps around the fruit, the system can precompute occlusion rankings for each fruit-direction pair during planning, allowing the robot to either select an unobstructed approach angle or preemptively manipulate blocking leaves. The main production challenge is that the method requires accurate instance segmentation of all plant organs as input—this is non-trivial in dense canopies and will require either high-quality 3D cameras (RGB-D, LiDAR) and robust segmentation models, or manual annotation at deployment sites. The compute cost is modest (runs on graph neural networks, not large transformers), and latency is acceptable for offline planning, but the system's performance depends heavily on segmentation accuracy, meaning errors propagate downstream.
Limitations and When Not to Use This
The evaluation is only on synthetic pepper plants, so generalization to real crops with natural occlusion patterns, variable lighting, and sensor noise remains unvalidated—real canopies have tangled leaves and deformable stems that may violate the static attachment assumptions. The method assumes each organ is already instance-segmented, which is a strong and expensive requirement; in practice, segmentation errors will propagate to the scene graph construction and occlusion predictions. The direction-conditioned framing assumes a fixed set of candidate approach directions, but doesn't address continuous direction spaces or handle cases where no unobstructed approach exists. The paper doesn't discuss how to handle dynamic occlusion (leaves moving, fruit shifting) or how the method scales to very dense canopies with hundreds of leaves per fruit.
Research Context
This work builds on prior research in robotic harvesting and scene understanding, where most prior approaches either use hand-engineered rules for leaf removal or treat occlusion prediction as a static geometry problem. It extends recent work on scene graphs for robotics manipulation by adding the direction-conditioning dimension, recognizing that occlusion is inherently viewpoint-dependent. The paper likely benchmarks against simpler baselines (point cloud convolutions, non-relational methods) and demonstrates that explicit relational reasoning via graph neural networks with attention improves performance. This opens a research direction toward more sophisticated manipulation planning that jointly optimizes for occlusion reasoning, geometric constraints, and approach feasibility.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::