"Predicting Future Actions of Reinforcement Learning Agents"

This paper evaluates and compares the effectiveness of future action and event prediction for three types of RL agents: explicitly planning,

implicitly planning, and non-planning.

📚 https://arxiv.org/abs/2410.22459

🎯 Original Problem:

Predicting future actions of deployed Reinforcement Learning (RL) agents is crucial for safe real-world applications. Current methods lack reliable ways to anticipate agent behavior, leading to potential safety risks in critical applications like autonomous vehicles.

-----

🔧 Solution in this Paper:

→ Introduces two prediction approaches:

- Inner State Approach: Uses agent's internal computations (plans, neuron activations)

- Simulation-based Approach: Runs agent through learned world model

→ Evaluates three agent types:

- Explicit planners (MuZero, Thinker)

- Implicit planners (Deep Repeated ConvLSTM)

- Non-planners (IMPALA)

→ Uses Sokoban environment with 5-step prediction horizon for both action sequences and event occurrence

-----

💡 Key Insights:

→ Explicit planning agents' internal plans are significantly more informative than other approaches

→ Inner state approach proves more robust to poor world models

→ High-quality world models enable excellent prediction in simulation-based approach

→ Model quality significantly impacts prediction accuracy

-----

📊 Results:

→ MuZero's action prediction accuracy jumps from 40% to 87% with inner state access

→ Simulation-based approach achieves highest accuracy with quality world models

→ Inner state approach shows 20-30% better robustness in degraded world model conditions