title: Multimodal Situational Safety
summary: Multimodal Large Language Models (MLLMs) are rapidly evolving, demonstrating
  impressive capabilities as multimodal assistants that interact with both humans
  and their environments. However, this increased sophistication introduces
  significant safety concerns. In this paper, we present the first evaluation and
  analysis of a novel safety challenge termed Multimodal Situational Safety,
  which explores how safety considerations vary based on the specific situation
  in which the user or agent is engaged. We argue that for an MLLM to respond
  safely, whether through language or action, it often needs to assess the safety
  implications of a language query within its corresponding visual context. To
  evaluate this capability, we develop the Multimodal Situational Safety
  benchmark (MSSBench) to assess the situational safety performance of current
  MLLMs. The dataset comprises 1,820 language query-image pairs, half of which
  the image context is safe, and the other half is unsafe. We also develop an
  evaluation framework that analyzes key safety aspects, including explicit
  safety reasoning, visual understanding, and, crucially, situational safety
  reasoning. Our findings reveal that current MLLMs struggle with this nuanced
  safety problem in the instruction-following setting and struggle to tackle
  these situational safety challenges all at once, highlighting a key area for
  future research. Furthermore, we develop multi-agent pipelines to coordinately
  solve safety challenges, which shows consistent improvement in safety over the
  original MLLM response. Code and data: mssbench.github.io.
paper_authors._id: 6707562501f0e95af7755b71
paper_authors.name: Kaiwen Zhou
paper_authors._id: 6707562501f0e95af7755b72
paper_authors.name: Chengzhi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:30:41.139Z
paper_authors._id: 65e71f6bcd3df9b0f6b2678b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e71f6bcd3df9b0f6b2678b/8mdex6eI80TGezot_GGI3.jpeg
paper_authors.fullname: Chengzhi Liu
paper_authors.type: user
paper_authors.user: LCZZZZ
paper_authors._id: 6707562501f0e95af7755b73
paper_authors.name: Xuandong Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:30:31.061Z
paper_authors._id: 6275a465597c70eb8949fce5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
paper_authors.fullname: Xuandong Zhao
paper_authors.type: user
paper_authors.user: Xuandong
paper_authors._id: 6707562501f0e95af7755b74
paper_authors.name: Anderson Compalas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:30:24.563Z
paper_authors._id: 670621c7bb23a4c6c7eea972
paper_authors.avatarUrl: /avatars/bdceb682f2b89a64316b1264bebb616a.svg
paper_authors.fullname: Anderson Compalas
paper_authors.type: user
paper_authors.user: acompalas
paper_authors._id: 6707562501f0e95af7755b75
paper_authors.name: Dawn Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:30:18.188Z
paper_authors._id: 6513179af60393414afd4140
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GAf-FulRRNF-X_Svew1Xj.jpeg
paper_authors.fullname: Xiaowei Song
paper_authors.type: user
paper_authors.user: dawnsong
paper_authors._id: 6707562501f0e95af7755b76
paper_authors.name: Xin Eric Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:30:11.281Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Multimodal Situational Safety
paper_summary: Multimodal Large Language Models (MLLMs) are rapidly evolving, demonstrating
  impressive capabilities as multimodal assistants that interact with both humans
  and their environments. However, this increased sophistication introduces
  significant safety concerns. In this paper, we present the first evaluation and
  analysis of a novel safety challenge termed Multimodal Situational Safety,
  which explores how safety considerations vary based on the specific situation
  in which the user or agent is engaged. We argue that for an MLLM to respond
  safely, whether through language or action, it often needs to assess the safety
  implications of a language query within its corresponding visual context. To
  evaluate this capability, we develop the Multimodal Situational Safety
  benchmark (MSSBench) to assess the situational safety performance of current
  MLLMs. The dataset comprises 1,820 language query-image pairs, half of which
  the image context is safe, and the other half is unsafe. We also develop an
  evaluation framework that analyzes key safety aspects, including explicit
  safety reasoning, visual understanding, and, crucially, situational safety
  reasoning. Our findings reveal that current MLLMs struggle with this nuanced
  safety problem in the instruction-following setting and struggle to tackle
  these situational safety challenges all at once, highlighting a key area for
  future research. Furthermore, we develop multi-agent pipelines to coordinately
  solve safety challenges, which shows consistent improvement in safety over the
  original MLLM response. Code and data: mssbench.github.io.
paper_ai_summary: A novel safety challenge, Multimodal Situational Safety, is evaluated through MSSBench, revealing MLLMs' struggles with situational safety reasoning in image contexts.
paper_ai_keywords: Multimodal Large Language Models MLLMs Multimodal Situational Safety MSSBench situational safety explicit safety reasoning visual understanding situational safety reasoning multi-agent pipelines