HP-Edit: A Human-Preference Post-Training Framework for Image Editing
| Authors | Fan Li et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.19406 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Common image editing tasks typically adopt powerful generative diffusion models as the leading paradigm for real-world content editing. Meanwhile, although reinforcement learning (RL) methods such as Diffusion-DPO and Flow-GRPO have further improved generation quality, efficiently applying Reinforcement Learning from Human Feedback (RLHF) to diffusion-based editing remains largely unexplored, due to a lack of scalable human-preference datasets and frameworks tailored to diverse editing needs. To fill this gap, we propose HP-Edit, a post-training framework for Human Preference-aligned Editing, and introduce RealPref-50K, a real-world dataset across eight common tasks and balancing common object editing. Specifically, HP-Edit leverages a small amount of human-preference scoring data and a pretrained visual large language model (VLM) to develop HP-Scorer--an automatic, human preference-aligned evaluator. We then use HP-Scorer both to efficiently build a scalable preference dataset and to serve as the reward function for post-training the editing model. We also introduce RealPref-Bench, a benchmark for evaluating real-world editing performance. Extensive experiments demonstrate that our approach significantly enhances models such as Qwen-Image-Edit-2509, aligning their outputs more closely with human preference.
Engineering Breakdown
Plain English
HP-Edit is a post-training framework that applies Reinforcement Learning from Human Feedback (RLHF) to image editing tasks powered by diffusion models. The authors created RealPref-50K, a dataset of 50,000 human preference annotations across eight common editing tasks, to address the lack of scalable preference data for editing workflows. Their framework achieves improved editing quality by leveraging human preference scoring data alongside diffusion models, without requiring full end-to-end human annotation. This work bridges a significant gap between recent RL advances in image generation (like Diffusion-DPO and Flow-GRPO) and practical application to real-world editing scenarios.
Core Technical Contribution
The core novelty is the first scalable RLHF framework specifically designed for diffusion-based image editing, moving beyond general image generation. The authors introduce two key technical contributions: (1) RealPref-50K, a systematically curated human-preference dataset spanning eight editing tasks with balanced object coverage, and (2) HP-Edit, a post-training method that efficiently applies preference learning without requiring expensive full-trajectory human annotations. Unlike prior RL work in generation that focuses on overall image quality, HP-Edit is tailored to diverse editing objectives like object replacement, style transfer, and local modifications. The framework uses a small amount of preference scoring data—not full preference pairs—making it more data-efficient than standard DPO approaches.
How It Works
HP-Edit operates as a post-training stage after a base diffusion model is trained. First, the framework collects human preference scores on editing results across the eight tasks in RealPref-50K, where annotators rate the quality of edits on dimensions like fidelity to the prompt, preservation of non-edited regions, and task-specific quality. The preference scores are then used to train a reward model or directly optimize the diffusion model using a preference-based loss (similar to DPO but adapted for editing). During inference, the trained model receives an image and text prompt specifying the edit, and the diffusion sampling process is guided by the preference-aligned parameters to produce edits that better match human expectations. The key architectural difference from generation is that the model must preserve the input image structure while applying targeted changes, requiring spatially-aware preference signals. The framework likely uses techniques from recent flow-based preference optimization (Flow-GRPO) adapted to the conditional editing setting where both the image and prompt inform the diffusion trajectory.
Production Impact
For production image editing systems, HP-Edit eliminates a major bottleneck: obtaining diverse, scalable preference data for fine-tuning without manual curation for each editing task. Engineers can now adopt a standard post-training recipe—collect preference scores on a held-out validation set of edits, run HP-Edit optimization, and deploy improved models without retraining from scratch. The RealPref-50K dataset (50K examples across eight tasks) is immediately usable as a benchmark or starting point, reducing time-to-deployment for new editing features. However, the trade-off is computational: preference-based training requires sampling multiple edit trajectories per prompt to estimate the preference signal, increasing GPU memory and wall-clock time compared to supervised fine-tuning. Integration requires a preference annotation pipeline (either synthetic or crowdsourced) and downstream A/B testing to verify improvements in actual user workflows, since preference models trained on one distribution may not transfer to live user preferences.
Limitations and When Not to Use This
HP-Edit assumes that human preference scores are reliable predictors of downstream user satisfaction, but real-world editing preferences are highly subjective and context-dependent—a preference learned from annotators on RealPref-50K may not generalize to users in different domains or with different aesthetic priorities. The framework is limited to the eight editing tasks in RealPref-50K (the abstract mentions these are common tasks, but specific coverage is unclear); extending to novel editing tasks would require collecting new preference data, limiting its zero-shot applicability. The paper doesn't discuss failure modes when the diffusion model lacks capacity to perform an edit well—preference learning may amplify errors rather than fix fundamental model limitations. Additionally, there is no analysis of computational overhead: how much slower is inference or training with HP-Edit compared to baseline diffusion models, and what is the minimum preference data threshold before improvements saturate? The framework likely requires careful tuning of the preference loss weight and sampling strategy, making it less plug-and-play than supervised fine-tuning.
Research Context
This work directly extends recent advances in preference-based diffusion model training—specifically Diffusion-DPO (which applies Direct Preference Optimization to generation) and Flow-GRPO (which uses flow matching and preference learning)—but applies them for the first time to the editing domain. It addresses a key limitation noted in the generative modeling literature: while RLHF has been transformative for language models and is emerging in vision generation, its application to conditional editing remains underexplored due to lack of datasets and task-specific frameworks. The RealPref-50K dataset contribution positions this work as a foundational resource for the community, similar to how other preference datasets (e.g., for LLMs) have catalyzed follow-on research. This opens a new research direction: understanding how to design preference learning objectives for spatially-conditioned generation tasks, which differ from unconditioned generation in that they require preserving input semantics while modifying targeted regions.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::