title: Keep Security! Benchmarking Security Policy Preservation in Large
    Language Model Contexts Against Indirect Attacks in Question Answering
summary: As Large Language Models (LLMs) are increasingly deployed in sensitive
  domains such as enterprise and government, ensuring that they adhere to
  user-defined security policies within context is critical-especially with
  respect to information non-disclosure. While prior LLM studies have focused on
  general safety and socially sensitive data, large-scale benchmarks for
  contextual security preservation against attacks remain lacking. To address
  this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating
  LLM adherence to contextual non-disclosure policies in question answering.
  Derived from realistic contexts, our dataset includes explicit policies and
  queries designed as direct and challenging indirect attacks seeking prohibited
  information. We evaluate 10 LLMs on our benchmark and reveal a significant
  vulnerability: many models violate user-defined policies and leak sensitive
  information. This failure is particularly severe against indirect attacks,
  highlighting a critical gap in current LLM safety alignment for sensitive
  applications. Our analysis reveals that while models can often identify the
  correct answer to a query, they struggle to incorporate policy constraints
  during generation. In contrast, they exhibit a partial ability to revise
  outputs when explicitly prompted. Our findings underscore the urgent need for
  more robust methods to guarantee contextual security.
paper_authors._id: 682eeb06720821973d643576
paper_authors.name: Hwan Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:18:42.752Z
paper_authors._id: 647c4a2692182942d7c2e698
paper_authors.avatarUrl: /avatars/bcddf5fe49aa092a2645f70812108348.svg
paper_authors.fullname: HWANCHANG
paper_authors.type: user
paper_authors.user: HwanChang0106
paper_authors._id: 682eeb06720821973d643577
paper_authors.name: Yumin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:44.617Z
paper_authors._id: 64aac6984135aae75f3b99c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yVR88SbW9TlJIrRHlJby-.jpeg
paper_authors.fullname: Yumin Kim
paper_authors.type: user
paper_authors.user: YuminKim
paper_authors._id: 682eeb06720821973d643578
paper_authors.name: Yonghyun Jun
paper_authors._id: 682eeb06720821973d643579
paper_authors.name: Hwanhee Lee
paper_title: Keep Security! Benchmarking Security Policy Preservation in Large
    Language Model Contexts Against Indirect Attacks in Question Answering
paper_summary: As Large Language Models (LLMs) are increasingly deployed in sensitive
  domains such as enterprise and government, ensuring that they adhere to
  user-defined security policies within context is critical-especially with
  respect to information non-disclosure. While prior LLM studies have focused on
  general safety and socially sensitive data, large-scale benchmarks for
  contextual security preservation against attacks remain lacking. To address
  this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating
  LLM adherence to contextual non-disclosure policies in question answering.
  Derived from realistic contexts, our dataset includes explicit policies and
  queries designed as direct and challenging indirect attacks seeking prohibited
  information. We evaluate 10 LLMs on our benchmark and reveal a significant
  vulnerability: many models violate user-defined policies and leak sensitive
  information. This failure is particularly severe against indirect attacks,
  highlighting a critical gap in current LLM safety alignment for sensitive
  applications. Our analysis reveals that while models can often identify the
  correct answer to a query, they struggle to incorporate policy constraints
  during generation. In contrast, they exhibit a partial ability to revise
  outputs when explicitly prompted. Our findings underscore the urgent need for
  more robust methods to guarantee contextual security.
paper_ai_summary: LLMs frequently violate contextual security policies by leaking sensitive information, particularly under indirect attacks, indicating a critical gap in current safety mechanisms.
paper_ai_keywords: Large Language Models (LLMs) security policies information non-disclosure CoPriva contextual security preservation question answering explicit policies indirect attacks virus policy constraints output revision