"TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning"

TriAdaptLoRA's triangular split and adaptive rank growth enhance LLM finetuning efficiency.

It dynamically adjusts trainable parameters in LLMs.

-----

https://arxiv.org/abs/2501.08008

Original Problem 🤔:

→ Fine-tuning LLMs is computationally expensive.

→ Existing Parameter-Efficient Fine-Tuning (PEFT) methods have limitations in rank adjustment and task adaptability.

-----

Solution in this Paper 💡:

→ TriAdaptLoRA introduces a triangular split of transformation matrices, dividing them into upper and lower triangular components. This maximizes parameter utilization and computational efficiency.

→ It employs a parameter importance metric based on normalized Frobenius norms. This simplifies rank adjustment compared to methods like AdaLoRA and IncreLoRA, reducing computational overhead.

→ It uses an adaptive rank-growth strategy guided by dynamic thresholds, enabling flexible parameter allocation during training. This improves upon fixed threshold methods by balancing parameter efficiency and model expressiveness.

-----

Key Insights from this Paper 🧠:

→ Triangular splitting of matrices allows for bidirectional parameter expansion, improving scalability and stability.

→ Normalized Frobenius norms offer an efficient way to assess the importance of incremental matrices.

→ Adaptive rank growth with dynamic thresholds enhances adaptability and reduces computational cost compared to fixed thresholds.

-----

Results 💯:

→ TriAdaptLoRA consistently outperforms existing PEFT methods like AdaLoRA and IncreLoRA on natural language understanding and generation tasks.

→ It achieves a performance improvement of approximately 0.44% on GLUE benchmark tasks compared to IncreLoRA, while substantially reducing computational overhead.

→ On SQUAD 2.0, the linear threshold mode improves EM and F1 scores by approximately 0.26% and 0.24%, respectively.