"RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation"

Sqrt > Division

RoRA fixes LoRA's scaling problem by using square root, making larger ranks actually useful.

LoRA's performance drops at rank 32, but RoRA keeps improving.

RoRA optimizes LoRA's scaling factor from α/r to α/√r, ensuring better performance with increasing rank sizes while maintaining gradient stability in LLM fine-tuning.

-----

https://arxiv.org/abs/2501.04315

🤔 Original Problem:

LoRA and DoRA show declining performance beyond rank 32, wasting computational resources without performance gains. This limitation stems from their scaling factor's inability to handle larger rank sizes effectively.

-----

🔧 Solution in this Paper:

→ RoRA introduces a new scaling factor α/√r instead of LoRA's α/r to maintain gradient stability across different rank sizes.

→ The mathematical analysis shows this change keeps gradient variance independent of rank, preventing performance degradation at higher ranks.

→ When rank is small, the scaling factor increases; when rank is large, it decreases, creating an optimal balance for gradient updates.

-----

💡 Key Insights:

→ Gradient stability is crucial for maintaining performance as rank size increases

→ The square root in the scaling factor prevents gradient explosion at higher ranks

→ The method works equally well for both uncompressed and pruned models

-----

📊 Results:

→ Outperforms LoRA by 6.5% and DoRA by 2.9% on LLaMA-7B for commonsense reasoning

→ Achieves 81.3% accuracy at rank 128, while others decline

→ Training time similar to LoRA (3h 45m for rank 8)

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/