LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation
| Authors | Jiazheng Xing et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20192 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Recent advances in diffusion models have significantly improved text-to-video generation, enabling personalized content creation with fine-grained control over both foreground and background elements. However, precise face-attribute alignment across subjects remains challenging, as existing methods lack explicit mechanisms to ensure intra-group consistency. Addressing this gap requires both explicit modeling strategies and face-attribute-aware data resources. We therefore propose LumosX, a framework that advances both data and model design. On the data side, a tailored collection pipeline orchestrates captions and visual cues from independent videos, while multimodal large language models (MLLMs) infer and assign subject-specific dependencies. These extracted relational priors impose a finer-grained structure that amplifies the expressive control of personalized video generation and enables the construction of a comprehensive benchmark. On the modeling side, Relational Self-Attention and Relational Cross-Attention intertwine position-aware embeddings with refined attention dynamics to inscribe explicit subject-attribute dependencies, enforcing disciplined intra-group cohesion and amplifying the separation between distinct subject clusters. Comprehensive evaluations on our benchmark demonstrate that LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject video generation. Code and models are available at https://jiazheng-xing.github.io/lumosx-home/.
Engineering Breakdown
Plain English
LumosX is a framework for text-to-video generation that solves the problem of maintaining consistent face attributes across different subjects in personalized video creation. The key innovation is a two-pronged approach: on the data side, they built a collection pipeline that uses multimodal large language models (MLLMs) to infer subject-specific dependencies and relational priors from independent videos; on the model side, they designed explicit mechanisms to enforce intra-group consistency for face attributes. This addresses a critical gap in existing diffusion-based video generation methods, which lack fine-grained control over preserving face-attribute alignment when generating content with multiple subjects. The framework enables more reliable personalized video content creation with better control over both foreground and background elements while maintaining visual coherence.
Core Technical Contribution
The paper's core novelty lies in explicitly modeling subject-specific relational priors through multimodal LLMs to enforce face-attribute consistency in text-to-video diffusion models. Rather than treating face attributes as implicit constraints learned during training, LumosX extracts these dependencies explicitly from data and embeds them into the generation process, creating a structured representation of how faces and their attributes should relate across subjects. The framework combines a tailored data collection pipeline that orchestrates captions and visual cues from independent videos with model-level mechanisms that use these extracted priors to guide generation. This dual innovation—better data representation through relational priors plus explicit architectural support for face-attribute alignment—represents a departure from prior end-to-end approaches that rely solely on implicit consistency learning.
How It Works
The system operates in two stages: data preparation and guided video generation. During data preparation, independent videos are processed to extract captions and visual cues; multimodal LLMs analyze these videos to infer subject-specific dependencies and relational priors (e.g., 'when subject A appears, subject B should have attribute X'). These relational priors are structured as constraints that capture intra-group consistency rules. During generation, a diffusion model conditions on text prompts and receives explicit guidance from the extracted relational priors, which enforce that face attributes remain aligned across different subjects throughout the video sequence. The model likely integrates these priors through attention mechanisms or conditioning modules that ensure the generated frames respect the subject-attribute relationships discovered during the data phase. This prevents attribute drift or inconsistency that would otherwise occur when generating multiple subjects with complex attribute dependencies.
Production Impact
For production video generation systems, LumosX would enable more reliable personalized content creation where face attributes must remain consistent—critical for deepfakes, avatar creation, social media content, or enterprise video personalization. The explicit relational priors extracted via MLLMs reduce hallucination and attribute inconsistency, eliminating manual post-processing or manual constraint specification that engineers currently perform. However, adoption requires significant upfront investment: building the tailored data collection pipeline, integrating an MLLM for dependency inference (adding inference latency and compute overhead), and retraining or fine-tuning diffusion models with the new constraint mechanism. Inference latency will increase due to MLLM dependency extraction and the additional guidance mechanisms, likely adding 10-30% computational overhead. For teams with strict latency requirements (e.g., real-time video conferencing), this trade-off may be prohibitive; for batch video generation or pre-production workflows, the consistency gains justify the cost.
Limitations and When Not to Use This
The paper assumes that relational priors extracted from independent videos generalize well to unseen subject combinations and novel attribute scenarios—an assumption that likely breaks down with out-of-distribution inputs. The approach is computationally expensive, requiring MLLM inference to extract priors for every new video scenario, which limits scalability and real-time applicability. The framework also depends heavily on the quality of MLLM inferences; if the MLLM misinterprets subject relationships or attributes, these errors propagate into the relational priors and corrupt video generation—the paper likely doesn't address robustness to MLLM failures. The method also doesn't solve the fundamental challenge of generating novel face attributes not seen during training, and it likely struggles with complex multi-subject scenes with many interdependent attributes. Additionally, the paper is incomplete (the abstract cuts off), so the actual results, ablations, and failure mode analysis are not available for evaluation.
Research Context
This work builds on recent advances in diffusion-based text-to-video generation (e.g., Imagen Video, Make-A-Video, Runway Gen-3) and extends them with explicit consistency mechanisms inspired by structured conditioning approaches in generative modeling. The relational prior extraction leverages the emerging capability of multimodal LLMs (GPT-4V, LLaVA, etc.) to reason about visual relationships, a direction gaining traction in vision-language model applications. The paper contributes to the broader research agenda of moving from implicit, end-to-end learning toward explicit, interpretable constraints in diffusion models—a trend seen in ControlNet, T2I-Adapter, and other structured conditioning work. This opens research directions in leveraging foundation models for constraint extraction, scaling relational prior representations to handle arbitrary subject numbers, and investigating whether such priors can transfer across domains without retraining.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::