Trans-LoRA enables data-free, lossless transfer of LoRA modules between different LLM architectures.
- Works for LoRA, DoRA, and Prompt Tuning PEFT methods
- Achieves up to 10% performance improvement in some cases
📚 https://arxiv.org/pdf/2405.17258
Original Problem 🔍:
Transferring LoRA modules between base models without access to original training data, crucial for cloud applications where providers can't retain client data.
-----
Solution in this Paper 🛠️:
• Trans-LoRA: Uses synthetic data for LoRA transfer
• Components:
- LLM-based synthetic data generator
- Discriminator for filtering synthetic data
- Knowledge distillation from source to target LoRA
• Operates with minimal seed examples (5 samples)
• Supports transfer across model families and PEFT types
-----
Key Insights from this Paper 💡:
• Synthetic data can effectively replace original training data for LoRA transfer
• Discriminator filtering crucial for matching source LoRA training distribution
• Lossless or improved transfer possible across different base models and PEFT methods
• Scalable performance with increased synthetic data generation
-----
Results 📊:
• Outperforms source LoRAs and target base models consistently
• Effective across Llama2 and Gemma model families•
• Supports continuous transfer through multiple models without degradation
