"Leveraging Graph-RAG and Prompt Engineering to Enhance LLM-Based Automated Requirement Traceability and Compliance Checks"

LLMs get smarter at checking software requirements by combining graph knowledge and structured thinking.

This paper combines Graph-RAG with advanced prompting techniques to automate compliance checking in software requirements, enhancing accuracy and reducing manual effort in regulated environments.

-----

https://arxiv.org/abs/2412.08593

🔍 Original Problem:

→ Software requirements in regulated sectors like finance and aerospace need strict compliance with higher-level standards, but manual checking is error-prone and time-consuming.

→ Current LLM solutions struggle with maintaining context across documents and often generate hallucinations during requirement validation.

-----

🛠️ Solution in this Paper:

→ The framework uses Graph-RAG to build a knowledge graph from regulatory documents, enabling precise retrieval of relevant content.

→ It employs Chain of Thought and Tree of Thought prompting to enhance LLM reasoning capabilities during compliance verification.

→ The system processes requirements through preprocessing, graph-based search, and structured prompting stages.

→ A similarity threshold balances between retrieval precision and coverage of relevant regulatory content.

-----

💡 Key Insights:

→ Graph-based retrieval outperforms traditional RAG in finding relevant regulatory content

→ Tree of Thoughts prompting achieves better reasoning than simpler prompting methods

→ Weaker models like GPT-4o-mini trade precision for higher recall in violation detection

-----

📊 Results:

→ Graph-RAG with Tree of Thoughts achieved 86.33% F1-score on broker dataset

→ Performance improved by 23.2% compared to baseline RAG methods

→ System maintained 87.93% F1-score on aerospace requirements