"SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization"

SymDPO makes multimodal AI actually look at images by speaking in symbols instead of words.

By replacing words with random symbols, SymDPO forces AI to truly understand what it sees.

SymDPO is a novel method that forces Large Multimodal Models (LMMs) to better utilize visual context in demonstrations by replacing text answers with random symbols. This addresses a critical issue where LMMs tend to overlook visual information and rely too heavily on text patterns when learning from examples.

-----

https://arxiv.org/abs/2411.11909

🤔 Original Problem:

LMMs often fail to effectively use visual information in multimodal demonstrations, instead defaulting to text pattern matching. Even when images are removed from demonstrations, model performance remains largely unchanged, indicating poor visual context integration.

-----

🔧 Solution in this Paper:

→ SymDPO replaces text answers in demonstrations with random symbols, forcing models to establish connections between images and symbols.

→ The model can only generate correct responses by understanding both the visual content and symbolic mapping, as the symbols themselves carry no semantic meaning.

→ A specialized data pipeline constructs symbolic preference datasets by replacing answers with contextually mismatched symbols to enforce visual-symbol alignment.

-----

💡 Key Insights:

→ Simply removing images from demonstrations barely impacts performance, revealing LMMs' over-reliance on text

→ Using semantically meaningless symbols forces models to actually process visual information

→ The approach works across different model architectures and benchmarks

-----

📊 Results:

→ Improved performance across 5 benchmarks including COCO Caption (+4.7% CIDEr) and VQAv2 (+0.5% accuracy)

→ Consistent gains across different shot counts (4, 8, 16) and model sizes (3B, 9B)

→ Outperforms other DPO methods like Video DPO and MIA-DPO