"Multimodal Latent Language Modeling with Next-Token Diffusion"

LatentLM makes multimodal AI speak one language by treating all data types as continuous tokens.

Next-token diffusion: The universal translator between discrete and continuous data in AI models.

LatentLM unifies continuous and discrete data processing in LLMs using causal Transformers and next-token diffusion, enabling seamless multimodal generation.

-----

https://arxiv.org/abs/2412.08635

🤔 Original Problem:

→ Current multimodal systems rely on separate modules for handling different data types, making end-to-end optimization difficult and causing information loss between components.

-----

🔧 Solution in this Paper:

→ LatentLM introduces a unified framework using causal Transformers to process both discrete and continuous data.

→ It employs VAE to compress continuous data into latent vectors.

→ Next-token diffusion is used to generate latent vectors autoregressively.

→ σ-VAE addresses variance collapse issues crucial for autoregressive modeling.

→ The system maintains a fixed variance in latent space for robust generation.

-----

💡 Key Insights:

→ Higher compression ratio while maintaining reconstruction quality

→ Simplified implementation by reusing LLM infrastructure

→ Unified interface for multimodal generation and understanding

→ Effective scaling with model size and training tokens

-----

📊 Results:

→ Outperforms Diffusion Transformers in image generation with fewer parameters

→ Achieves 10x fewer decoding steps compared to VALL-E 2 in text-to-speech

→ Shows 2.84x throughput improvement at batch size 256

→ Surpasses Transfusion in multimodal tasks with FID score of 14.54