SAFE: Multitask Failure Detection for Vision-Language-Action Models
While vision-language-action models (VLAs) have shown promising robotic
behaviors across a diverse set of manipulation tasks, they achieve limited
success rates when deployed on novel tasks out-of-the-box. To allow these
policies to safely interact with their environments, we need a failure detector
that gives a timely alert such that the robot can stop, backtrack, or ask for
help. However, existing failure detectors are trained and tested only on one or
a few specific tasks, while VLAs require the detector to generalize and detect
failures also in unseen tasks and novel environments. In this paper, we
introduce the multitask failure detection problem and propose SAFE, a failure
detector for generalist robot policies such as VLAs. We analyze the VLA feature
space and find that VLAs have sufficient high-level knowledge about task
success and failure, which is generic across different tasks. Based on this
insight, we design SAFE to learn from VLA internal features and predict a
single scalar indicating the likelihood of task failure. SAFE is trained on
both successful and failed rollouts, and is evaluated on unseen tasks. SAFE is
compatible with different policy architectures. We test it on OpenVLA, pi_0,
and pi_0-FAST in both simulated and real-world environments extensively. We
compare SAFE with diverse baselines and show that SAFE achieves
state-of-the-art failure detection performance and the best trade-off between
accuracy and detection time using conformal prediction. More qualitative
results can be found at https://vla-safe.github.io/.
684a3d639b38e1e5a33a686d
Qiao Gu
684a3d639b38e1e5a33a686e
Yuanliang Ju
claimed_verified
2025-06-17T21:13:29.343Z
646f4cf46b3df773a2c93fd0
https://cdn-avatars.huggingface.co/v1/production/uploads/646f4cf46b3df773a2c93fd0/8SGp5O6tmbd26Oxte9Gpy.jpeg
Yuanliang Ju
user
Yuanliang
684a3d639b38e1e5a33a686f
Shengxiang Sun
claimed_verified
2025-06-13T07:41:49.417Z
64e1ccaee8b6f3f3bae535a4
/avatars/23aae0cca24aa7190ab290bb2e2187d9.svg
Owen Sun
user
theOman
684a3d639b38e1e5a33a6870
Igor Gilitschenski
684a3d639b38e1e5a33a6871
Haruki Nishimura
684a3d639b38e1e5a33a6872
Masha Itkina
684a3d639b38e1e5a33a6873
Florian Shkurti
SAFE: Multitask Failure Detection for Vision-Language-Action Models
While vision-language-action models (VLAs) have shown promising robotic
behaviors across a diverse set of manipulation tasks, they achieve limited
success rates when deployed on novel tasks out-of-the-box. To allow these
policies to safely interact with their environments, we need a failure detector
that gives a timely alert such that the robot can stop, backtrack, or ask for
help. However, existing failure detectors are trained and tested only on one or
a few specific tasks, while VLAs require the detector to generalize and detect
failures also in unseen tasks and novel environments. In this paper, we
introduce the multitask failure detection problem and propose SAFE, a failure
detector for generalist robot policies such as VLAs. We analyze the VLA feature
space and find that VLAs have sufficient high-level knowledge about task
success and failure, which is generic across different tasks. Based on this
insight, we design SAFE to learn from VLA internal features and predict a
single scalar indicating the likelihood of task failure. SAFE is trained on
both successful and failed rollouts, and is evaluated on unseen tasks. SAFE is
compatible with different policy architectures. We test it on OpenVLA, pi_0,
and pi_0-FAST in both simulated and real-world environments extensively. We
compare SAFE with diverse baselines and show that SAFE achieves
state-of-the-art failure detection performance and the best trade-off between
accuracy and detection time using conformal prediction. More qualitative
results can be found at https://vla-safe.github.io/.
SAFE is a failure detector for vision-language-action models that generalizes to unseen tasks by learning from high-level internal features of the models.
vision-language-action models VLAs multitask failure detection failure detector feature space high-level knowledge scalar prediction rollout conformal prediction