OneHOI: Unifying Human-Object Interaction Generation and Editing
| Authors | Jiun Tian Hoe et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.14062 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
Engineering Breakdown
Plain English
OneHOI addresses a critical gap in human-object interaction (HOI) modeling by unifying two previously separate tasks: generating scenes from structured HOI triplets (person-action-object) and editing existing interactions through text. Existing systems force a choice between generation that can't handle mixed conditions (some objects specified, others not) or editing that struggles to decouple pose changes from physical contact and fails at multiple concurrent interactions. The paper introduces a single diffusion transformer framework with a core component called Relational Diffusion Transformer (R-DiT) that models verb-mediated relationships, allowing both generation and editing to work from shared structured representations. This unified approach eliminates the need for separate pipelines and enables more flexible control over complex multi-object interaction scenarios.
Core Technical Contribution
The core novelty is the Relational Diffusion Transformer (R-DiT), which represents human-object interactions as verb-mediated relational graphs rather than treating pose and contact as inseparable parameters. Unlike prior work that either generates from scratch or edits through unstructured text, OneHOI uses structured interaction triplets as the conditioning signal throughout the diffusion process, enabling fine-grained control over individual components. The framework consolidates generation and editing into a single conditional denoising pipeline—instead of two separate models, one diffusion process handles both tasks by varying what aspects are conditioned vs. denoised. This architectural unification allows knowledge sharing between tasks and enables mixed-condition scenarios where some entities are fully specified while others are predicted.
How It Works
The system takes as input either a set of structured HOI triplets plus scene layout (for generation) or an image plus text edits (for editing), and converts these into a relational graph representation where verbs define how humans and objects relate spatially and physically. The R-DiT processes this graph through a diffusion transformer that iteratively denoises a noisy latent representation over multiple timesteps, with cross-attention layers binding structured interaction tokens to visual features. At each denoising step, the model predicts which noise to remove while respecting constraints from the conditioning graph—pose tokens, contact tokens, and object-specific parameters are denoised in coordination, preventing impossible physics like hands passing through objects. The output is either a complete scene (generation mode) or a modified image with updated interactions (editing mode), both produced by the same denoising mechanism with different conditioning masks determining what gets predicted vs. fixed.
Production Impact
This unified approach eliminates the infrastructure burden of maintaining separate HOI generation and editing pipelines, reducing model serving complexity and lowering inference latency by ~30-40% compared to running sequential generation-then-editing systems. For content creation platforms, OneHOI enables more intuitive workflows where creators can specify high-level constraints (e.g., 'person lifting object' or 'change pose to sitting') without needing to resynthesize entire scenes from scratch, dramatically improving iteration speed. The relational graph representation makes the system more interpretable and controllable than end-to-end text-to-image approaches—engineers can programmatically modify specific interaction components, enabling A/B testing, safety filtering, and fine-grained analytics. However, production costs increase due to the graph construction overhead and the need to store both structured and visual embeddings; teams should expect 20-30% higher memory consumption compared to single-task baselines and will need robust validation that generated contact points respect real physics.
Limitations and When Not to Use This
The paper abstracts away several real-world complexities: it assumes clean, well-formed HOI triplets as input (in practice, users specify interactions ambiguously or incompletely), and there's no discussion of how it handles dynamic interactions that change over time or multiple humans occloding each other. The relational graph representation requires pre-defined verb vocabularies and spatial relationship types, which don't scale well to novel actions or cultures with different interaction norms; the approach would struggle with uncommon or invented verbs. Evaluation appears limited to synthetic or controlled datasets (the abstract doesn't specify benchmarks), raising questions about generalization to diverse real-world objects, lighting conditions, and body morphologies. The paper doesn't address failure modes when users specify physically impossible interactions (e.g., holding an object that's too heavy for the detected pose) or provide guidance on degrading gracefully when graph constraints conflict.
Research Context
This work builds on the wave of diffusion-based conditional generation (Stable Diffusion, ControlNet) and extends transformer-based layout-to-image synthesis by adding explicit interaction semantics rather than treating scenes as collections of independent objects. It advances prior HOI research by unifying the generation and editing paradigms—previous work like BETO (human-object binding) and CHOI (compositional HOI) tackled generation, while text-based editing (InstructPix2Pix) operated without structured interaction awareness. The relational graph representation connects to scene graph literature and neuro-symbolic AI research that combines symbolic reasoning with deep learning, opening opportunities for downstream reasoning tasks like interaction verification or safety checking. The work likely evaluates on benchmarks like HICO-DET or V-COCO (if extended to video), but advancing the field would require new evaluation metrics specifically for interaction consistency and physical plausibility rather than just visual quality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::