Athena: Retrieval-augmented Legal Judgment Prediction with Large Language Models
RAG-powered framework teaches LLMs legal expertise through smart case retrieval
RAG-powered framework teaches LLMs legal expertise through smart case retrieval
Original Problem 🔍:
Legal judgment prediction (LJP) tasks struggle with LLMs' hallucinations and lack of domain-specific knowledge.
Solution in this Paper 🧠:
• Athena: A novel framework for LJP using retrieval-augmented generation (RAG)
• Constructs a knowledge base of accusations with descriptions and examples
• Employs semantic retrieval to find relevant accusation candidates
• Utilizes LLMs for final judgment prediction with retrieved context
Key Insights from this Paper 💡:
• RAG significantly enhances LLMs' performance in LJP tasks
• Query rewriting improves retrieval performance
• Optimal in-context window size balances retrieval and LLM comprehension
• Performance varies across accusation types, with challenges in similar and multi-accusation cases
Results 📊:
• Athena outperforms baseline, zero-shot chain-of-thought, and legal syllogism prompting methods
• Achieves up to 95% accuracy with GPT-4o and optimal parameter tuning
• Improves performance for smaller models like GPT-3.5-turbo when using RAG
• Hit Rate increases significantly with query rewriting, especially beneficial for low-capacity LLMs
