Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-07 with 1 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ziyun Zeng et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2605.06535 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
In recent years, open-source efforts like Senorita-2M have propelled video editing toward natural language instruction. However, current publicly available datasets predominantly focus on local editing or style transfer, which largely preserve the original scene structure and are easier to scale. In contrast, Background Replacement, a task central to creative applications such as film production and advertising, requires synthesizing entirely new, temporally consistent scenes while maintaining accurate foreground-background interactions, making large-scale data generation significantly more challenging. Consequently, this complex task remains largely underexplored due to a scarcity of high-quality training data. This gap is evident in poorly performing state-of-the-art models, e.g., Kiwi-Edit, because the primary open-source dataset that contains this task, i.e., OpenVE-3M, frequently produces static, unnatural backgrounds. In this paper, we trace this quality degradation to a lack of precise background guidance during data synthesis. Accordingly, we design a scalable pipeline that generates foreground and background guidance in a decoupled manner with strict quality filtering. Building on this pipeline, we introduce Sparkle, a dataset of ~140K video pairs spanning five common background-change themes, alongside Sparkle-Bench, the largest evaluation benchmark tailored for background replacement to date. Experiments demonstrate that our dataset and the model trained on it achieve substantially better performance than all existing baselines on both OpenVE-Bench and Sparkle-Bench. Our proposed dataset, benchmark, and model are fully open-sourced at https://showlab.github.io/Sparkle/.
Engineering Breakdown
Plain English
Sparkle addresses video background replacement guided by natural language instructions, a task that's much harder than existing video editing work because it requires synthesizing entirely new backgrounds while keeping the foreground stable and temporally consistent across frames. The paper tackles this by proposing a decoupled guidance approach and addressing the core blocker: the near-total absence of large-scale, high-quality training data for this specific task, which has left current models (like Kiwi-E) performing poorly.
Key Engineering Insight
The critical insight is that background replacement requires decoupling foreground preservation from background synthesis during guidance—you can't use the same control mechanism for both without introducing artifacts. This architectural separation is essential because the guidance signals for 'keep the person stable' and 'generate a new beach background' operate at fundamentally different scales and timelines.
Why It Matters for Engineers
Production video editing workflows (film, advertising, content creation) need reliable background replacement that respects temporal coherence across frames and preserves actor/subject details. Current public models fail at this, forcing studios to use expensive manual compositing or proprietary tools. A working open-source solution directly solves a real production bottleneck.
Research Context
Prior work (Senorita-2M and similar) focused on local edits and style transfer—tasks that preserve scene structure and scale easily. Background replacement is structurally different and data-hungry, making it largely unexplored despite being central to creative workflows. This paper fills that gap by proposing both the technical approach (decoupled guidance) and implicitly, the need for better datasets to make the task viable at scale.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::