"Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2502.03738

The common approach in Vision Transformer models compresses images into patches, reducing computational cost but potentially losing visual information. This paper addresses the problem of information loss due to image patchification in vision models.

This paper proposes to reduce the patch size, even to single pixels, to minimize information loss and improve model performance.

-----

📌 Smaller patch sizes in Vision Transformers access finer image details. This enhances visual information fidelity directly at the input level. Models benefit from richer representations without complex architectural changes.

📌 Computational cost shifts from model parameters to sequence length. Mamba architecture's linear complexity becomes crucial. Pixel-level tokenization is now feasible for high resolution images, unlocking new scaling dimensions.

📌 Decoder heads become less critical for tasks like semantic segmentation. High fidelity encoders with pixel tokens suffice. This simplifies architecture and suggests a move towards encoder-only visual models.

----------

Methods Explored in this Paper 🔧:

→ The paper investigates the impact of patch size by conducting experiments with Vision Transformer and Mamba based architectures.

→ They systematically reduced the patch size from 16x16 down to 1x1 pixel.

→ This reduction in patch size increases the input sequence length significantly, up to 50,176 tokens for a 224x224 image.

→ Experiments were performed on ImageNet-1k classification, ADE20k semantic segmentation, and COCO object detection/instance segmentation tasks.

→ Both Vision Transformer and Adventurer architectures were used to ensure findings are generalizable.

-----

Key Insights 💡:

→ A scaling law in patchification is observed.

→ Smaller patch sizes consistently improve model performance across various vision tasks, input resolutions, and architectures.

→ Patchification is identified as a compromise for computational efficiency, not a necessity for effective vision models.

→ Reducing patch size unlocks crucial visual information previously lost through compression.

→ Task specific decoder heads become less critical for dense prediction tasks when using smaller patch sizes.

-----

Results 📊:

→ ImageNet-1k classification accuracy improved from 82.6% to 84.6% on Adventurer-Base model by reducing patch size to 1x1 on 224x224 images.

→ On ADE20k semantic segmentation, mIoU improved consistently as patch size decreased, reaching 46.8% mIoU with Adventurer-Base and 2x2 patch size without a decoder head.

→ COCO object detection APb improved from 44.7% to 48.7% on Adventurer-Tiny and from 44.1% to 50.3% on Adventurer-Base by reducing patch size.