"NVILA: Efficient Frontier Visual Language Models"

Scale up visual processing, then compress it - that's how NVILA saves GPU power

NVILA introduces an efficient family of visual language models that optimize both accuracy and efficiency through a "scale-then-compress" approach, reducing training costs by 4.5x while maintaining competitive performance with leading models. The paper addresses the critical challenge of making visual language models more resource-efficient without compromising their capabilities.

-----

https://arxiv.org/abs/2412.04468

🤔 Original Problem:

→ Current visual language models are expensive to train, requiring up to 400 GPU days for a 7B model.

→ Fine-tuning requires over 64GB GPU memory, beyond most consumer hardware.

→ Deployment on edge devices is constrained by limited computational resources.

-----

⚡ Solution in this Paper:

→ NVILA uses a "scale-then-compress" strategy that first increases spatial and temporal resolution, then compresses visual tokens.

→ The model employs Dynamic-S2 for adaptive image processing with varying aspect ratios.

→ It introduces DeltaLoss for dataset pruning, reducing training data while maintaining performance.

→ FP8 training accelerates computation while preserving accuracy.

→ Specialized quantization techniques optimize both vision encoder and language model components.

-----

🔍 Key Insights:

→ Higher resolution processing followed by token compression achieves better accuracy-efficiency trade-off

→ Temporal averaging effectively reduces video token redundancy

→ Dataset pruning can halve training data without significant performance loss

-----

📊 Results:

→ 4.5x reduction in training costs

→ 3.4x decrease in fine-tuning memory usage

→ 1.6-2.2x improvement in pre-filling latency

→ 1.2-2.8x boost in decoding throughput