title: A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI
    Responses
summary: Large Language Models (LLMs) are vulnerable to
  jailbreaksx2013methods to elicit harmful or generally impermissible
  outputs. Safety measures are developed and assessed on their effectiveness at
  defending against jailbreak attacks, indicating a belief that safety is
  equivalent to robustness. We assert that current defense mechanisms, such as
  output filters and alignment fine-tuning, are, and will remain, fundamentally
  insufficient for ensuring model safety. These defenses fail to address risks
  arising from dual-intent queries and the ability to composite innocuous outputs
  to achieve harmful goals. To address this critical gap, we introduce an
  information-theoretic threat model called inferential adversaries who exploit
  impermissible information leakage from model outputs to achieve malicious
  goals. We distinguish these from commonly studied security adversaries who only
  seek to force victim models to generate specific impermissible outputs. We
  demonstrate the feasibility of automating inferential adversaries through
  question decomposition and response aggregation. To provide safety guarantees,
  we define an information censorship criterion for censorship mechanisms,
  bounding the leakage of impermissible information. We propose a defense
  mechanism which ensures this bound and reveal an intrinsic safety-utility
  trade-off. Our work provides the first theoretically grounded understanding of
  the requirements for releasing safe LLMs and the utility costs involved.
paper_authors._id: 6686b9c3434e14e5a1ee2182
paper_authors.name: David Glukhov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:24:05.040Z
paper_authors._id: 653be9f07773b2d575ba3f7c
paper_authors.avatarUrl: /avatars/2f605f3bcd8680bbce88ebd316d838ec.svg
paper_authors.fullname: David Glukhov
paper_authors.type: user
paper_authors.user: WhereIsExit
paper_authors._id: 6686b9c3434e14e5a1ee2183
paper_authors.name: Ziwen Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:24:12.680Z
paper_authors._id: 62ed40fe9b538d02b451c211
paper_authors.avatarUrl: /avatars/d5486b050fd63ade348158e64a3e5ff7.svg
paper_authors.fullname: Han
paper_authors.type: user
paper_authors.user: Ziwen
paper_authors._id: 6686b9c3434e14e5a1ee2184
paper_authors.name: Ilia Shumailov
paper_authors._id: 6686b9c3434e14e5a1ee2185
paper_authors.name: Vardan Papyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:24:26.211Z
paper_authors._id: 6581f96f0fb25b9aa270856e
paper_authors.avatarUrl: /avatars/5502416aed6352ecf794252735656a4b.svg
paper_authors.fullname: Vardan Papyan
paper_authors.type: user
paper_authors.user: vardanpapyan
paper_authors._id: 6686b9c3434e14e5a1ee2186
paper_authors.name: Nicolas Papernot
paper_title: A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI
    Responses
paper_summary: Large Language Models (LLMs) are vulnerable to
  jailbreaksx2013methods to elicit harmful or generally impermissible
  outputs. Safety measures are developed and assessed on their effectiveness at
  defending against jailbreak attacks, indicating a belief that safety is
  equivalent to robustness. We assert that current defense mechanisms, such as
  output filters and alignment fine-tuning, are, and will remain, fundamentally
  insufficient for ensuring model safety. These defenses fail to address risks
  arising from dual-intent queries and the ability to composite innocuous outputs
  to achieve harmful goals. To address this critical gap, we introduce an
  information-theoretic threat model called inferential adversaries who exploit
  impermissible information leakage from model outputs to achieve malicious
  goals. We distinguish these from commonly studied security adversaries who only
  seek to force victim models to generate specific impermissible outputs. We
  demonstrate the feasibility of automating inferential adversaries through
  question decomposition and response aggregation. To provide safety guarantees,
  we define an information censorship criterion for censorship mechanisms,
  bounding the leakage of impermissible information. We propose a defense
  mechanism which ensures this bound and reveal an intrinsic safety-utility
  trade-off. Our work provides the first theoretically grounded understanding of
  the requirements for releasing safe LLMs and the utility costs involved.
paper_ai_keywords: jailbreaks dual-intent queries output filters alignment fine-tuning inferential adversaries information-theoretic threat model information censorship criterion intrinsic safety-utility trade-off Large Language Models (LLMs)