"Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge"

A specialized LLM that makes rare disease knowledge accessible to everyone

Context-aware fine-tuning meets medical expertise in this rare disease focused LLM

https://arxiv.org/abs/2411.02657

🎯 Original Problem:

Rare diseases like Ehlers-Danlos Syndrome (EDS) suffer from fragmented information and delayed diagnosis. Traditional LLMs struggle with specialized medical domains due to limited data and expertise availability.

-----

🔧 Solution in this Paper:

→ Built Zebra-Llama, a context-aware LLM specifically for EDS using Llama-3.1-8B-Instruct as base model

→ Implemented novel context-aware fine-tuning with Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA)

→ Created comprehensive training data from PubMed, Reddit, and Inspire patient forums in Question-Context-Answer format

→ Developed sophisticated classification mechanism using embedding vectors and cosine similarity to distinguish EDS vs non-EDS queries

→ Built high-precision Retrieval Augmented Generation (RAG) pipeline with Pinecone vector database of 50,000+ indexed entries

-----

💡 Key Insights:

→ Domain-specific LLMs can effectively democratize rare disease knowledge

→ Context-aware fine-tuning significantly improves model's ability to utilize retrieved information

→ Balancing precision and recall in domain specificity is crucial for medical applications

-----

📊 Results:

→ Thoroughness improved from 70.1% to 77.5%

→ Accuracy increased from 78.8% to 83.0%

→ Clarity enhanced from 72.0% to 74.7%

→ Citation reliability jumped from 52.3% to 70.6%