New smart weight-compression technique has arrived to reduce your GPU VRAM requirement.
Squeeze more parameters into your GPU by compressing the wasteful parts of floating-point numbers
NeuZip compresses neural networks by exploiting the low entropy nature of floating-point exponents
📚 https://arxiv.org/abs/2410.20650
🎯 Original Problem:
Training and deploying large neural networks is severely constrained by GPU memory limitations. While model sizes have grown 100x since 2017, GPU memory has only increased 2.5x (from 32GB to 80GB), creating a critical bottleneck for scaling neural networks.
-----
🔧 Solution in this Paper:
→ Introduces NeuZip - a novel compression scheme that exploits low entropy in neural network parameters' exponent bits
→ Compresses exponent bits losslessly using asymmetric numeral system (ANS) while keeping sign and mantissa bits unchanged
→ Implements layer-by-layer compression/decompression during training to avoid creating large buffers
→ Compatible with activation checkpointing for additional memory savings
→ For inference, introduces lossy compression by truncating mantissa bits while controlling relative weight changes
-----
💡 Key Insights:
→ Neural network parameters tend to concentrate around zero, making exponent bits highly compressible
→ Exponent bits carry only ~3 bits of information despite using 8 bits of storage
→ Layer-wise compression enables training without ever fully decompressing the entire network
→ Inference tasks can tolerate more aggressive lossy compression compared to training
-----
📊 Results:
→ Reduces Llama-3 8B training memory from 31GB to <16GB with no performance loss
→ Enables training 13B parameter models on consumer GPUs (<20GB memory)
→ For inference, achieves >50% memory reduction while maintaining near-lossless performance
→ Outperforms QLoRA and other quantization methods in memory-performance trade-off
Share this post