LoraMap: Harnessing the Power of LoRA Connections

LoraMap teaches language models to combine different types of reasoning for improved fact-checking

Connecting reasoning LoRAs through LoraMap boosts fact-checking performance in language models.

📚 https://arxiv.org/pdf/2408.16264v1

Original Problem 🔍:

Existing LoRA composition methods lack attention to connections between multiple LoRAs, limiting their effectiveness in fact-checking tasks.

-----

Solution in this Paper 🧠:

• Creates three reasoning datasets for fact-checking: DifferenceCoT, EntityCoT, and CorrectClaim

• Fine-tunes individual LoRAs on these datasets

• Introduces LoraMap: learns connections between LoRAs instead of linear sum

• LoraMap concatenates matrices of multiple reasoning LoRAs

• Inserts trainable mapping matrices (Amap and Bmap) between them

• Freezes original LoRAs to maintain specialized reasoning capabilities

• Fine-tunes only Amap and Bmap matrices

-----

Key Insights from this Paper 💡:

• LoraMap outperforms LoraHub and LoraConcat with fewer parameters

• Connecting multiple specialized LoRAs enhances fact-checking performance

• Flexible scaling of trainable parameters based on model size and task requirements

• Potential applications in other NLP tasks beyond fact-checking

-----

Results 📊:

• LoraMap outperforms LoraHub and LoraConcat on COVID-Fact dataset

• Flan-T5-large: LoraMap (0.22M parameters) achieves superior performance

• Flan-T5-xxl: LoraMap (4.4M) outperforms LoraConcat (56M) with fewer parameters

• Macro-f1 scores: LoraMap (0.8239) > LoraConcat (0.8126) > LoraHub (0.6145)