Embodied Red Teaming for Auditing Robotic Foundation Models
Language-conditioned robot models have the potential to enable robots to
perform a wide range of tasks based on natural language instructions. However,
assessing their safety and effectiveness remains challenging because it is
difficult to test all the different ways a single task can be phrased. Current
benchmarks have two key limitations: they rely on a limited set of
human-generated instructions, missing many challenging cases, and focus only on
task performance without assessing safety, such as avoiding damage. To address
these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method
that generates diverse and challenging instructions to test these models. ERT
uses automated red teaming techniques with Vision Language Models (VLMs) to
create contextually grounded, difficult instructions. Experimental results show
that state-of-the-art language-conditioned robot models fail or behave unsafely
on ERT-generated instructions, underscoring the shortcomings of current
benchmarks in evaluating real-world performance and safety. Code and videos are
available at: https://s-karnik.github.io/embodied-red-team-project-page.
67ab837b02329ca8f809ceae
Sathwik Karnik
claimed_verified
2025-02-12T09:16:48.921Z
67ab79ead3f12c4887867022
/avatars/d8e1dd4afabd4c2d04a73712a610cea6.svg
Sathwik Karnik
user
S-Karnik
67ab837b02329ca8f809ceaf
Zhang-Wei Hong
67ab837b02329ca8f809ceb0
Nishant Abhangi
67ab837b02329ca8f809ceb1
Yen-Chen Lin
67ab837b02329ca8f809ceb2
Tsun-Hsuan Wang
67ab837b02329ca8f809ceb3
Christophe Dupuy
67ab837b02329ca8f809ceb4
Rahul Gupta
67ab837b02329ca8f809ceb5
Pulkit Agrawal
Embodied Red Teaming for Auditing Robotic Foundation Models
Language-conditioned robot models have the potential to enable robots to
perform a wide range of tasks based on natural language instructions. However,
assessing their safety and effectiveness remains challenging because it is
difficult to test all the different ways a single task can be phrased. Current
benchmarks have two key limitations: they rely on a limited set of
human-generated instructions, missing many challenging cases, and focus only on
task performance without assessing safety, such as avoiding damage. To address
these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method
that generates diverse and challenging instructions to test these models. ERT
uses automated red teaming techniques with Vision Language Models (VLMs) to
create contextually grounded, difficult instructions. Experimental results show
that state-of-the-art language-conditioned robot models fail or behave unsafely
on ERT-generated instructions, underscoring the shortcomings of current
benchmarks in evaluating real-world performance and safety. Code and videos are
available at: https://s-karnik.github.io/embodied-red-team-project-page.
Embodied Red Teaming uses automated red teaming with Vision Language Models to generate diverse, challenging instructions for evaluating the safety and effectiveness of language-conditioned robot models.
Vision Language Models embodied red teaming contextually grounded state-of-the-art language-conditioned robot models