"CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers"
Below podcast on this paper is generated with Google's Illuminate.
https://arxiv.org/abs/2502.06527
Personalized video generation faces challenges in maintaining consistent subject identity and temporal coherence across video frames. This paper introduces CustomVideoX to solve these problems by dynamically adapting a video diffusion transformer using a 3D reference attention mechanism.
This approach enables zero-shot customized video generation with improved consistency and quality.
-----
📌 3D Reference Attention in CustomVideoX is architecturally significant. It tightly integrates reference information into Video Diffusion Transformer (VDiT). This contrasts with external adapter methods for cleaner feature fusion.
📌 Time-Aware Attention Bias (TAB) provides critical temporal modulation. The parabolic bias schedule smartly addresses the evolving need for reference guidance across diffusion steps.
📌 Entity Region-Aware Enhancement (ERAE) achieves semantic-aware refinement. By latent space operations, it boosts entity focus without restrictive spatial constraints.
----------
Methods Explored in this Paper 🔧:
→ CustomVideoX uses 3D Reference Attention. This mechanism facilitates direct interaction between a reference image and all frames of a video. It operates within the Video Diffusion Transformer (VDiT) framework. This eliminates the need for separate spatial and temporal attention stages.
→ Time-Aware Attention Bias (TAB) modulates the influence of reference image features across denoising steps. TAB employs a parabolic temporal mask. This dynamically weights reference features, starting low, increasing in mid-phases, and decreasing in final stages. This enhances temporal coherence.
→ Entity Region-Aware Enhancement (ERAE) module focuses on key entity regions. It refines attention bias by aligning highly activated regions of entity tokens with reference features. This is achieved by adjusting attention bias based on a threshold.
-----
Key Insights 💡:
→ 3D Reference Attention enables efficient and effective interaction between reference images and video frames. It improves both spatial and temporal consistency in generated videos.
→ Time-Aware Attention Bias dynamically manages the contribution of reference features during denoising. This leads to better temporal coherence and visual quality. It captures both overall structure and fine details.
→ Entity Region-Aware Enhancement adaptively emphasizes key entity regions. This enhances subject fidelity without compromising the diversity of generated content.
-----
Results 📊:
→ On VideoBench, CustomVideoX achieves a CLIP-T score of 33.38, CLIP-I of 90.26, DINO-I of 91.49 and Temporal Consistency (T.Cons) of 97.26.
→ CustomVideoX outperforms the second best method MS-Diffusion by 0.52% in CLIP-T and 1.72% in DINO-I on VideoBench.
→ CustomVideoX achieves the highest Temporal Consistency (T.Cons) score of 97.26 on VideoBench, demonstrating improved video quality.