"MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs"

This survey bridges the gap between scattered MLLM benchmarks and organized evaluation methods

Organizes and analyzes evaluation benchmarks for Multimodal LLMs (MLLMs)

-----

https://arxiv.org/abs/2411.15296

🔍 Methods used in this Paper:

→ The paper presents a hierarchical taxonomy of MLLM evaluation benchmarks across foundation capabilities, model behavior, and extended applications.

→ It outlines benchmark construction methods, including data collection and QA pair annotation processes.

→ The survey introduces three evaluation approaches: human-based, LLM-based, and script-based assessment.

→ It provides insights into future benchmark directions, focusing on capability taxonomy and task-oriented evaluation.

-----

💡 Key Insights:

→ MLLMs struggle with fine-grained perception tasks and visual mathematics

→ Open-source models are increasingly matching closed-source performance

→ Complex localization and structural relationships remain challenging

→ High-resolution data significantly improves object recognition and text understanding