"Aligning Large Language Models for Faithful Integrity Against Opposing Argument"

LLMs now know when to stick to their guns and when to change their mind

AFICE framework helps LLMs maintain their correct stance against opposing arguments while admitting mistakes when wrong, using confidence estimation and preference optimization.

-----

https://arxiv.org/abs/2501.01336

🤔 Original Problem:

LLMs often change their correct answers when faced with opposing arguments from users, compromising their reliability in conversations. Even advanced models like GPT-4 struggle to maintain accurate stances when challenged.

-----

🔧 Solution in this Paper:

→ The AFICE framework uses Bilateral Confidence Estimation (BCE) to measure how certain an LLM is about its responses

→ BCE analyzes internal model states during generation and adjusts confidence using probability ratios

→ The framework samples multiple responses using beam sampling and truncates early tokens for efficiency

→ It creates preference datasets based on model confidence levels

→ Uses Direct Preference Optimization to fine-tune the model for maintaining faithful responses

-----

💡 Key Insights:

→ Model confidence should match response certainty for faithful interactions

→ Early token truncation (first 60 tokens) maintains accuracy while reducing computation

→ Combining internal states and probability ratios provides better confidence estimation

→ Flexible response strategies based on confidence levels improve conversation quality

-----

📊 Results:

→ AFICE outperforms baselines across Mathematics, Logic, Commonsense and Generic tasks

→ Achieves 74.2% accuracy on LLaMA-3 vs 59.8% baseline

→ Shows 67.2% accuracy on Vicuna vs 50.2% baseline

→ Demonstrates lower Expected Calibration Error compared to other methods

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/