Flexora: Flexible Low Rank Adaptation for Large Language Models

Replace LoRA with Flexora (flexible low-rank adaptation) 🔥

Flexora auto-selects which LLM layers to fine-tune, cutting training costs. Think precision pruning for LLMs - that's what Flexora brings to the table

Flexora's flexible approach to LoRA fine-tuning yields superior results and reduces training parameters by up to 50% 🤯

Introduces adaptive layer selection for LoRA

https://arxiv.org/abs/2408.10774

Key Insights 💡:

• Selective layer fine-tuning can significantly reduce overfitting in LLMs

• Automatic and flexible layer selection is crucial for optimal performance across tasks

• Framing layer selection as a hyperparameter optimization problem yields superior results

• Combining Flexora with other LoRA variants further enhances performance

Solution in this Paper 🛠️:

• Frames layer selection as a hyperparameter optimization (HPO) problem

• Uses unrolled differentiation (UD) to solve the HPO problem efficiently

• Implements a two-stage process:

- Flexible layer selection stage: Optimizes hyperparameters to identify crucial layers

- Fine-tuning stage: Retrains selected LoRA parameters from scratch

• Allows for both automatic and flexible selection of layers to fine-tune

• Integrates well with other LoRA variants like DoRA and rsLoRA

Results 📊:

• Outperforms existing baselines across multiple common sense reasoning tasks

• Average accuracy improvement:

- +7.21% on Llama3-8B

- +8.33% on ChatGLM3-6B

- +1.98% on Mistral-7B-v0.1

• Demonstrates strong generalization and scalability across different LLMs

• Effectively mitigates overfitting in various downstream tasks.