"LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents"

LossAgent introduces an LLM-based framework that enables optimization of image processing models using any objective, even non-differentiable ones.

It dynamically adjusts loss weights based on feedback from external evaluators, making previously unusable optimization objectives now possible.

-----

https://arxiv.org/abs/2412.04090

🔍 Original Problem:

→ Traditional image processing models are limited by differentiable loss functions, preventing the use of advanced quality metrics and human feedback for optimization.

→ Complex perceptual metrics and text-based feedback cannot be directly used for training, limiting model improvement.

-----

🛠️ Solution in this Paper:

→ LossAgent uses an LLM to interpret feedback from any optimization objective and convert it into usable loss weights.

→ The system maintains a repository of standard loss functions and dynamically adjusts their weights during training.

→ A three-part prompt engineering strategy guides the LLM: system prompts define goals, historical prompts provide context, and customized prompts ensure consistent output format.

→ The agent analyzes model performance and optimization trajectory to make informed weight adjustments.

-----

💡 Key Insights:

→ LLMs can effectively bridge non-differentiable objectives with trainable loss functions

→ Historical optimization trajectories help prevent LLM hallucination

→ Format standardization significantly improves LLM output reliability (99.87% success rate)

-----

📊 Results:

→ Outperformed baseline methods across multiple image processing tasks

→ Achieved better NIQE scores (4.08 vs 4.23 baseline)

→ Successfully handled textual feedback with comparable performance to numerical metrics