"TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models"

Smart prompt adjustments make AI vision models bulletproof against attacks

TAPT ( Test-Time Adversarial Prompt Tuning), proposed in this paper, introduces a novel test-time defense method that dynamically tunes prompts during inference to protect Vision-Language Models against adversarial attacks, while maintaining performance on clean samples without task-specific training.

-----

https://arxiv.org/abs/2411.13136

🔍 Original Problem:

Vision-Language Models like CLIP are vulnerable to small adversarial perturbations in input images, significantly degrading their inference performance. Traditional defenses require costly adversarial training, while existing prompt tuning methods need task-specific training and struggle with generalization.

-----

🛠️ Solution in this Paper:

→ TAPT operates by generating multiple augmented views of test samples and selecting views with low entropy in their averaged predictions

→ It optimizes defensive prompts by minimizing multi-view entropy across selected views

→ The method aligns adversarial-clean distributions using pre-computed statistics from public datasets

→ TAPT works during inference without requiring any task-specific training or annotations

→ The system dynamically adapts prompts for each test sample to ensure robust predictions

-----

💡 Key Insights:

→ Test-time prompt tuning can effectively defend against adversarial attacks without expensive training

→ Multi-view entropy helps identify reliable augmented views of test samples

→ Aligning embeddings with pre-computed statistics improves robustness while maintaining clean accuracy

-----

📊 Results:

→ Improves zero-shot adversarial robustness by 48.9% against AutoAttack

→ Achieves 36.6% robustness improvement with ViT-B/16 backbone

→ Outperforms existing methods across 11 benchmark datasets