title: Code-as-Monitor: Constraint-aware Visual Programming for Reactive and
    Proactive Robotic Failure Detection
summary: Automatic detection and prevention of open-set failures are crucial in
  closed-loop robotic systems. Recent studies often struggle to simultaneously
  identify unexpected failures reactively after they occur and prevent
  foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a
  novel paradigm leveraging the vision-language model (VLM) for both open-set
  reactive and proactive failure detection. The core of our method is to
  formulate both tasks as a unified set of spatio-temporal constraint
  satisfaction problems and use VLM-generated code to evaluate them for real-time
  monitoring. To enhance the accuracy and efficiency of monitoring, we further
  introduce constraint elements that abstract constraint-related entities or
  their parts into compact geometric elements. This approach offers greater
  generality, simplifies tracking, and facilitates constraint-aware visual
  programming by leveraging these elements as visual prompts. Experiments show
  that CaM achieves a 28.7% higher success rate and reduces execution time by
  31.8% under severe disturbances compared to baselines across three simulators
  and a real-world setting. Moreover, CaM can be integrated with open-loop
  control policies to form closed-loop systems, enabling long-horizon tasks in
  cluttered scenes with dynamic environments.
paper_authors._id: 6752744f8681347ff4207fc2
paper_authors.name: Enshen Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:08:56.444Z
paper_authors._id: 63f08dc79cf89c9ed1bb89cd
paper_authors.avatarUrl: /avatars/37290358ad00bbd752f519cfdec02f3e.svg
paper_authors.fullname: Zhoues
paper_authors.type: user
paper_authors.user: Zhoues
paper_authors._id: 6752744f8681347ff4207fc3
paper_authors.name: Qi Su
paper_authors._id: 6752744f8681347ff4207fc4
paper_authors.name: Cheng Chi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:13.512Z
paper_authors._id: 650bf938677f9e45963d672e
paper_authors.avatarUrl: /avatars/7d4159067b5005a3a635e36b26b7b239.svg
paper_authors.fullname: Cheng Chi
paper_authors.type: user
paper_authors.user: ChuckChi
paper_authors._id: 6752744f8681347ff4207fc5
paper_authors.name: Zhizheng Zhang
paper_authors._id: 6752744f8681347ff4207fc6
paper_authors.name: Zhongyuan Wang
paper_authors._id: 6752744f8681347ff4207fc7
paper_authors.name: Tiejun Huang
paper_authors._id: 6752744f8681347ff4207fc8
paper_authors.name: Lu Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:56:27.453Z
paper_authors._id: 65b722dbe02a17f0f8d1cc6b
paper_authors.avatarUrl: /avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
paper_authors.fullname: Lu Sheng
paper_authors.type: user
paper_authors.user: lsheng2024
paper_authors._id: 6752744f8681347ff4207fc9
paper_authors.name: He Wang
paper_title: Code-as-Monitor: Constraint-aware Visual Programming for Reactive and
    Proactive Robotic Failure Detection
paper_summary: Automatic detection and prevention of open-set failures are crucial in
  closed-loop robotic systems. Recent studies often struggle to simultaneously
  identify unexpected failures reactively after they occur and prevent
  foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a
  novel paradigm leveraging the vision-language model (VLM) for both open-set
  reactive and proactive failure detection. The core of our method is to
  formulate both tasks as a unified set of spatio-temporal constraint
  satisfaction problems and use VLM-generated code to evaluate them for real-time
  monitoring. To enhance the accuracy and efficiency of monitoring, we further
  introduce constraint elements that abstract constraint-related entities or
  their parts into compact geometric elements. This approach offers greater
  generality, simplifies tracking, and facilitates constraint-aware visual
  programming by leveraging these elements as visual prompts. Experiments show
  that CaM achieves a 28.7% higher success rate and reduces execution time by
  31.8% under severe disturbances compared to baselines across three simulators
  and a real-world setting. Moreover, CaM can be integrated with open-loop
  control policies to form closed-loop systems, enabling long-horizon tasks in
  cluttered scenes with dynamic environments.
paper_ai_summary: Code-as-Monitor (CaM) uses vision-language models to address both reactive and proactive open-set failure detection in closed-loop robotic systems by formulating them as spatio-temporal constraint satisfaction problems.
paper_ai_keywords: vision-language model VLM spatio-temporal constraint satisfaction problems code-as-monitor constraint elements geometric elements open-set failures closed-loop systems