title: Cross-Modality Safety Alignment
summary: As Artificial General Intelligence (AGI) becomes increasingly integrated into
  various facets of human life, ensuring the safety and ethical alignment of such
  systems is paramount. Previous studies primarily focus on single-modality
  threats, which may not suffice given the integrated and complex nature of
  cross-modality interactions. We introduce a novel safety alignment challenge
  called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety
  alignment. Specifically, it considers cases where single modalities are safe
  independently but could potentially lead to unsafe or unethical outputs when
  combined. To empirically investigate this problem, we developed the SIUO, a
  cross-modality benchmark encompassing 9 critical safety domains, such as
  self-harm, illegal activities, and privacy violations. Our findings reveal
  substantial safety vulnerabilities in both closed- and open-source LVLMs, such
  as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably
  interpret and respond to complex, real-world scenarios.
paper_authors._id: 667aa83869c3e097d596de29
paper_authors.name: Siyin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T13:34:23.735Z
paper_authors._id: 64c3c631e77ea9f28111172a
paper_authors.avatarUrl: /avatars/495dbb73b69c399bae780da3118e332f.svg
paper_authors.fullname: Siyin Wang
paper_authors.type: user
paper_authors.user: sinwang
paper_authors._id: 667aa83869c3e097d596de2a
paper_authors.name: Xingsong Ye
paper_authors._id: 667aa83869c3e097d596de2b
paper_authors.name: Qinyuan Cheng
paper_authors._id: 667aa83869c3e097d596de2c
paper_authors.name: Junwen Duan
paper_authors._id: 667aa83869c3e097d596de2d
paper_authors.name: Shimin Li
paper_authors._id: 667aa83869c3e097d596de2e
paper_authors.name: Jinlan Fu
paper_authors._id: 667aa83869c3e097d596de2f
paper_authors.name: Xipeng Qiu
paper_authors._id: 667aa83869c3e097d596de30
paper_authors.name: Xuanjing Huang
paper_title: Cross-Modality Safety Alignment
paper_summary: As Artificial General Intelligence (AGI) becomes increasingly integrated into
  various facets of human life, ensuring the safety and ethical alignment of such
  systems is paramount. Previous studies primarily focus on single-modality
  threats, which may not suffice given the integrated and complex nature of
  cross-modality interactions. We introduce a novel safety alignment challenge
  called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety
  alignment. Specifically, it considers cases where single modalities are safe
  independently but could potentially lead to unsafe or unethical outputs when
  combined. To empirically investigate this problem, we developed the SIUO, a
  cross-modality benchmark encompassing 9 critical safety domains, such as
  self-harm, illegal activities, and privacy violations. Our findings reveal
  substantial safety vulnerabilities in both closed- and open-source LVLMs, such
  as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably
  interpret and respond to complex, real-world scenarios.
paper_ai_keywords: Safe Inputs but Unsafe Output (SIUO) cross-modality safety alignment cross-modality benchmark self-harm illegal activities privacy violations LVLMs GPT-4V LLaVA complex, real-world scenarios