"IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models"

The paper addresses the limitations of current Text-to-Image (T2I) evaluation methods by introducing imaginee, a new benchmark to assess T2I models more comprehensively across diverse and challenging tasks. This benchmark aims to better reflect human perception and evaluate the progress of T2I models towards becoming general-purpose models.

-----

Paper - https://arxiv.org/abs/2501.13920

Original Problem 😕:

→ Existing evaluation methods for Text-to-Image models are too simple.

→ They fail to accurately reflect human perception of image quality.

→ There is a significant gap between evaluation results and human intuitive judgments.

-----

Solution in this Paper 💡:

→ This paper introduces \imaginee, a comprehensive evaluation framework for Text-to-Image models.

→ \imaginee uses five diverse domains to rigorously assess model capabilities.

→ These domains include structured output generation, realism and physical consistency, specific domain generation, challenging scenarios, and multi-style creation.

→ For quantitative evaluation, \imaginee employs multiple methods.

→ These methods are CLIPScore, HPSv2, Aesthetic Score, GPT-4o based scoring, and Human evaluation.

→ Human evaluation uses the same criteria as GPT-4o, focusing on aesthetics, realism, safety, and text-image matching.

-----

Key Insights from this Paper 🧐:

→ Text-to-Image models are expanding beyond basic image generation.

→ They are showing potential in structured output and specialized domains.

→ Evaluating Text-to-Image models requires diverse and challenging tasks.

→ Comprehensive benchmarks like \imaginee are crucial for assessing true progress.

→ Combining automated metrics with human evaluation provides a robust assessment.

-----

Results 📊:

→ Evaluated six representative Text-to-Image models: FLUX.1, Ideogram2.0, Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng.

→ The evaluation used \imaginee across five domains and multiple quantitative metrics including human evaluation.

→ This study also compared automated scoring methods with human ratings to check validity.