MaKTO (Multi-agent KTO) outperforms GPT-4o by 23.0% and two-stage RL agents by 10.9% in win rate.
MaKTO learns social strategy through language game immersion.
This paper introduces Multi-agent Kahneman & Tversky’s Optimization (MaKTO) to train LLMs for strategic interaction in language games. MaKTO enables models to learn via in-context interaction, unlike traditional decoupled decision-making approaches.
-----
Paper - https://arxiv.org/abs/2501.14225
Original Problem: 🤔:
→ Current AI agents in language games often separate decision-making from language generation.
→ This decoupling limits generalization and strategic depth in complex social interactions.
→ Existing methods fail to fully integrate language and action as described by Wittgenstein's Language Game Theory.
-----
Solution in this Paper: 💡:
→ The paper proposes Multi-agent Kahneman & Tversky’s Optimization (MaKTO).
→ MaKTO trains LLMs through direct interaction in a multi-agent game environment.
→ Behavior cloning (BC) initializes the model using expert game data and strategies.
→ MaKTO employs Kahneman & Tversky Optimization (KTO) for fine-tuning decision-making.
→ It uses multi-agent gameplay with diverse models to prevent strategy fixation.
→ Stepwise preference selection refines actions using heuristic, voting, and verifier-based methods.
-----
Key Insights from this Paper: 🔑:
→ Integrating language and decision-making is crucial for advanced AI agents.
→ Wittgenstein's Language Game Theory inspires a more unified approach to AI development.
→ Social deduction games like Werewolf are excellent testbeds for strategic language agents.
→ Multi-agent interaction during training enhances robustness and generalization.
→ Stepwise feedback provides more granular optimization than win-loss outcomes alone.
-----
Results: 🏆:
→ MaKTO achieves a 61% average win rate in 9-player Werewolf games.
→ MaKTO (Multi-agent KTO) outperforms GPT-4o by 23.0%
→ MaKTO achieves a 60% win rate against human expert players.
→ In Turing tests, MaKTO shows only 48.9% detectability, indicating human-like gameplay.
