"Embodied Red Teaming for Auditing Robotic Foundation Models"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2411.18676

The paper addresses the problem that current benchmarks for evaluating language-conditioned robot models are insufficient for real-world use because they fail to capture the diverse ways users might phrase instructions and do not adequately assess safety.

This paper introduces Embodied Red Teaming (ERT), a novel evaluation method using Vision Language Models (VLMs) to automatically generate diverse and challenging instructions, iteratively refined with robot feedback, to audit robot models effectively.

-----

📌 ERT automates red teaming by using VLMs to generate diverse instructions grounded in robot environments. This bypasses the limitations of human-generated benchmarks and scales testing for real-world deployment.

📌 The iterative in-context refinement in ERT leverages failure feedback to dynamically create challenging instructions. This adaptive approach is crucial for uncovering nuanced vulnerabilities in language-conditioned robot models.

📌 By using VLMs, ERT inherently understands visual context for instruction generation. This vision grounding enables the creation of robot-relevant instructions that are both feasible and semantically rich, unlike pure language model approaches.

----------

Methods Explored in this Paper 🔧:

→ Embodied Red Teaming (ERT) is introduced as an automated evaluation method.

→ ERT uses Vision Language Models (VLMs) to generate contextually grounded instructions.

→ It iteratively refines instructions using in-context learning and feedback from the robot's execution.

→ The process starts with a task description and environment image.

→ A VLM, specifically GPT-4o in this paper, generates N instructions.

→ Instructions are filtered to be feasible for the robot in the given environment.

→ The robot attempts to execute these instructions. Success or failure is recorded.

→ Instructions causing failure are used as examples to refine the VLM's subsequent instruction generation, increasing the challenge.

→ To ensure diversity, ERT samples M sets of instructions and selects the most diverse set based on CLIP embeddings.

-----

Key Insights 💡:

→ Current benchmarks for language-conditioned robots are limited.

→ These benchmarks do not represent the diversity of real-world instructions.

→ State-of-the-art robot models perform well on existing benchmarks but struggle with ERT-generated instructions.

→ This performance drop highlights a lack of generalization in current robot models.

→ ERT exposes vulnerabilities in instruction generalization and safety that existing benchmarks miss.

→ Instructions causing failure on one robot model often cause failure on others, indicating a shared vulnerability.

-----

Results 📊:

→ 3D-Diffuser model's success rate on CALVIN training instructions is 90%. On ERT-generated instructions, it drops to 53%.

→ 3D-Diffuser model's success rate on RLBench training instructions is 79%. On ERT-generated instructions, it drastically falls to 3%.

→ ERT achieves significantly higher BLEU diversity in generated instructions compared to existing benchmarks and rephrased instructions.

→ OpenVLA model, a larger 7B parameter model, shows a success rate of 76% on SimplerEnv benchmark instructions, but only 30.8% on ERT-generated instructions.