"QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2502.05178

The challenge is to create a unified model for both visual understanding and generation, but current visual tokenizers are optimized for only one of these tasks. This paper introduces Quantized Language-Image Pretraining (QLIP) to bridge this gap.

This paper proposes QLIP, a visual tokenizer trained with both reconstruction and language-image alignment objectives. QLIP uses a two-stage training process and dynamic loss weighting to achieve state-of-the-art performance in both areas.

-----

📌 QLIP pioneers text-aligned visual tokenization. It uniquely integrates visual reconstruction with semantic understanding within the tokenizer itself. This enables superior zero-shot image classification at 74.3% accuracy and improved generation FID at 15.29.

📌 The two-stage training of QLIP is crucial. Stage one balances contrastive alignment and reconstruction using dynamic loss weights. Stage two optimizes reconstruction with perceptual and adversarial losses. This overcomes memory limits and gradient conflicts.

📌 QLIP's unified multimodal model, UM3, demonstrates versatility. It handles text-only, image-to-text, and text-to-image tasks within a single architecture. UM3 achieves comparable performance to specialized models, showcasing QLIP's effectiveness.

----------

Methods Explored in this Paper 🔧:

→ QLIP is a visual tokenizer based on Binary Spherical Quantization. It is trained as an autoencoder to reconstruct images into discrete visual tokens.

→ Simultaneously, QLIP is trained with a contrastive language-image objective. This aligns visual tokens with text embeddings, enhancing semantic representation.

→ A two-stage training pipeline is employed. Stage 1 trains QLIP with both alignment and reconstruction losses using a memory-efficient Transformer. Stage 2 fine-tunes the quantizer and decoder with reconstruction losses, after freezing the visual encoder and dropping the text encoder. This improves reconstruction quality by using perceptual and adversarial losses without memory limitations.

→ To balance the competing alignment and reconstruction objectives, an automated weighting scheme is introduced. Loss terms are weighted by the inverse of their post-hoc loss values. This addresses the differing gradient magnitudes and convergence rates of the two objectives.

-----

Key Insights 💡:

→ Balancing reconstruction and alignment objectives during visual tokenizer training is crucial for achieving strong performance in both understanding and generation.

→ A two-stage training process effectively addresses the challenges of large batch contrastive learning and memory-intensive reconstruction losses.

→ Initializing the visual encoder with Masked Image Modeling or CLIP pre-training significantly accelerates convergence and improves performance.

-----

Results 📊:

→ QLIP-B achieves 74.3% zero-shot classification accuracy on ImageNet, comparable to CLIP models while also providing visual tokenization.

→ QLIP-B achieves a reconstruction rFID of 3.21, demonstrating state-of-the-art reconstruction quality for a semantically aligned tokenizer.

→ In text-conditioned image generation on MS-COCO 30k, LlamaGen with QLIP-B achieves a generation FID of 15.29, outperforming LlamaGen with VQGAN (15.68).