"QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture"

QuArch introduces a dataset of 1,500 expert-validated questions to test LLMs' understanding of computer architecture concepts.

Addressing the critical gap in hardware engineering AI tools.

-----

https://arxiv.org/abs/2501.01892

🤔 Original Problem:

Hardware engineering lags in AI adoption due to LLMs' limited understanding of computer architecture concepts and lack of specialized evaluation datasets.

-----

🔧 Solution in this Paper:

→ Created Archipedia, a comprehensive corpus exceeding 1 billion tokens synthesizing 50 years of computer architecture knowledge

→ Generated questions using commercial LLMs from Archipedia content

→ Implemented multi-tier validation with domain experts and LLM assistance

→ Covered 13 core areas including processor design, memory systems, and interconnection networks

→ Built an evaluation framework to assess LLM performance on architecture concepts

-----

💡 Key Insights:

→ LLMs struggle most with memory systems and interconnection networks

→ Domain-specific datasets are crucial for developing AI expertise in hardware

→ Expert validation is essential for quality question-answer pairs

→ Fine-tuning improves small model performance significantly

-----

📊 Results:

→ Best closed-source model: 84% accuracy

→ Top small open-source model: 72% accuracy

→ Fine-tuning improves small models by 5.4%-8.3%

→ Performance comparable to MMLU (88%) but better than GPQA (54%)

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/