"Improving Video Generation with Human Feedback"

Human-preferred videos, generated by AI

The paper proposes is to improve video generation by better aligning with human preferences using a novel reward maximization method. This method enhances video quality by directly optimizing for human feedback.

-----

Paper - https://arxiv.org/abs/2501.13918

Original Problem 🤔:

→ Current video generation models often fail to align with human aesthetic preferences and satisfaction.

→ Existing methods struggle to effectively incorporate nuanced human feedback into the training process.

-----

Solution in this Paper 💡:

→ This paper introduces a novel approach called Flow Direct Preference Optimization (Flow-DPO).

→ Flow-DPO is a likelihood-based reward maximization method specifically designed for aligning video generation with human preferences.

→ It leverages a flow-based reward model to directly optimize the video generation policy based on pairwise human preference data.

→ The method also incorporates a technique called Reward-Weighted Regression (RWR) to further refine the alignment process.

→ Flow-DPO aims to overcome limitations of traditional reinforcement learning methods in video generation.

-----

Key Insights from this Paper 🧐:

→ Directly optimizing for human preferences using pairwise comparison data is crucial for improving video generation quality.

→ Flow-based reward models offer a more effective way to capture complex human preferences compared to traditional scalar reward models.

→ Likelihood-based optimization methods like Flow-DPO can lead to more stable and efficient alignment in video generation.

-----

Results ✨:

→ The proposed Flow-DPO method achieves a 81.3% win rate against DPO in human preference evaluations.

→ Flow-DPO demonstrates a 7.1% improvement in preference accuracy compared to standard DPO.

→ Experiments show that Flow-DPO outperforms existing alignment methods in generating videos that are more preferred by humans.