"Can Modern LLMs Act as Agent Cores in Radiology~Environments?"

Can AI read X-rays like a pro? RadABench puts LLMs through medical school.

This paper introduces RadABench, a comprehensive framework to evaluate LLMs as agent cores in radiology environments. It assesses LLMs' ability to interact with specialized tools for complex medical imaging tasks.

-----

https://arxiv.org/abs/2412.09529

🔍 Original Problem:

Existing LLMs lack evaluation in complex radiology tasks involving tool interaction and multi-step reasoning.

-----

🧠 Solution in this Paper:

→ RadABench-Data: A synthetic dataset with 2,200 patient records, 24,200 QA pairs, and 10 tool categories.

→ RadABench-EvalPlat: An evaluation platform simulating diverse radiology toolsets and conditions.

→ Three-stage architecture: Task Decomposition, Tool Selection & Execution, and Response Generation.

→ Dynamic tool set simulation strategy to reflect various real-world clinical scenarios.

-----

💡 Key Insights from this Paper:

→ Closed-source models outperform open-source models in most metrics

→ LLMs struggle with complex tool understanding and multi-round information synthesis

→ Tool incomplete hallucinations and input/output format issues persist

→ Performance degrades with increasing task complexity and incomplete tool availability

-----

📊 Results:

→ GPT-4o and Claude-3.5-Sonnet show best performance in chain planning and execution

→ Gemini-1.5-Pro achieves 0.720 Optimal Tool Score

→ Claude-3.5-sonnet reaches 0.795 Execution Completion Rate in SNN Regular condition

→ All LLMs demonstrate low BLEU, F1, and ROUGE scores (rarely exceeding 0.35) in final responses