"Leveraging Graph-RAG and Prompt Engineering to Enhance LLM-Based Automated Requirement Traceability and Compliance Checks"

Graph-RAG turns messy software requirements into organized knowledge graphs for precise compliance checking.

Graph-RAG and advanced prompting techniques enhance automated requirement traceability in software development, improving compliance verification through better context retrieval and reasoning capabilities.

-----

https://arxiv.org/abs/2412.08593

🔍 Original Problem:

→ Traditional LLMs struggle with maintaining context across extensive software requirement documents, leading to incomplete compliance checks and hallucinations.

→ Current methods lack precision in retrieving relevant regulatory content and often miss nuanced domain-specific requirements.

-----

⚡ Solution in this Paper:

→ The paper introduces a framework combining Graph-RAG with Chain of Thought and Tree of Thought prompting.

→ Graph-RAG constructs a graph-based text index from regulatory articles, enabling precise content retrieval through entity relationships.

→ Advanced prompting techniques break down complex compliance checks into logical steps, exploring multiple reasoning paths simultaneously.

→ The system was tested on two real-world cases: an Iranian broker application and NASA's X-38 fault-tolerant system.

-----

🎯 Key Insights:

→ Graph structure significantly improves retrieval accuracy compared to traditional RAG

→ Smaller text chunks provide better recall but require more processing calls

→ Dynamic weighting system helps prioritize relevant sub-communities in the graph

→ Human oversight remains crucial for nuanced regulatory interpretations

-----

📊 Results:

→ Graph-RAG with Tree of Thought achieved 86.33% F1-score on broker dataset

→ Performance improved to 87.93% F1-score on aerospace requirements

→ Outperformed baseline RAG methods by 15-20% in compliance detection