"Flowing from Words to Pixels: A Framework for Cross-Modality Evolution"

CrossFlow eliminates noise and conditioning by directly evolving between modality distributions.

Directly transform text to images without complex conditioning mechanisms

CrossFlow enables direct evolution between different modalities using flow matching, eliminating the need for noise distribution and conditioning mechanisms.

https://arxiv.org/abs/2412.15213

Original Problem 🤔:

→ Current diffusion and flow matching models rely on mapping from noise to target distribution, requiring complex conditioning mechanisms for cross-modal tasks

→ This approach is computationally expensive and architecturally complex, especially for text-to-image generation

-----

Solution in this Paper 🛠️:

→ CrossFlow introduces a framework that directly maps between source and target modality distributions

→ A Variational Encoder converts source data to match target shape while maintaining semantic information

→ The model uses a binary indicator for Classifier-Free Guidance instead of traditional conditioning

→ Flow matching then evolves the encoded source directly into the target distribution

-----

Key Insights 💡:

→ Direct modality evolution is possible without noise distribution or conditioning

→ CrossFlow scales better with model size and training steps than traditional approaches

→ The framework enables semantic latent arithmetic operations

→ A unified architecture works across multiple cross-modal tasks

-----

Results 📊:

→ Outperforms standard flow matching baseline with 10.13 vs 10.79 FID score

→ Achieves 9.63 FID-30K on COCO dataset using only 0.95B parameters

→ Matches state-of-the-art performance on image captioning (BLEU-4: 36.4)

→ Comparable depth estimation results on KITTI (AbsRel: 0.062) and NYUv2 (AbsRel: 0.103)