"FastVLM: Efficient Vision Encoding for Vision Language Models"

FastVLM makes vision-language models 85x faster by smartly reducing visual tokens without losing quality.

FastVLM introduces a hybrid vision encoder that processes high-resolution images efficiently while reducing visual tokens, making vision-language models faster and more efficient.

-----

https://arxiv.org/abs/2412.13303

🔍 Original Problem:

Vision Language Models struggle with high-resolution images due to inefficient visual encoders like ViTs that generate too many tokens and have high encoding latency.

-----

🛠️ Solution in this Paper:

→ FastVLM uses FastViTHD, a novel hybrid vision encoder combining convolutional and transformer layers in a 5-stage architecture.

→ The model strategically downsamples input tensors by 64x, significantly reducing the number of tokens processed by self-attention layers.

→ FastViTHD generates 4x fewer tokens than traditional architectures while maintaining high-quality visual representations.

→ The architecture eliminates the need for additional token pruning by naturally achieving optimal token count through input scaling.

-----

💡 Key Insights:

→ Hybrid architectures (convolution + transformer) are more efficient than pure transformer models for vision encoding

→ Vision encoder latency dominates total processing time at high resolutions

→ Optimal performance requires balancing image resolution, token count, and LLM size

-----

📊 Results:

→ 3.2x faster time-to-first-token compared to previous methods

→ 85x faster than LLaVA-OneVision at 1152x1152 resolution

→ 3.4x smaller vision encoder while maintaining comparable performance

→ Achieves state-of-the-art accuracy-latency trade-off on M1 MacBook Pro