Keep Security! Benchmarking Security Policy Preservation in Large
  Language Model Contexts Against Indirect Attacks in Question Answering
As Large Language Models (LLMs) are increasingly deployed in sensitive
domains such as enterprise and government, ensuring that they adhere to
user-defined security policies within context is critical-especially with
respect to information non-disclosure. While prior LLM studies have focused on
general safety and socially sensitive data, large-scale benchmarks for
contextual security preservation against attacks remain lacking. To address
this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating
LLM adherence to contextual non-disclosure policies in question answering.
Derived from realistic contexts, our dataset includes explicit policies and
queries designed as direct and challenging indirect attacks seeking prohibited
information. We evaluate 10 LLMs on our benchmark and reveal a significant
vulnerability: many models violate user-defined policies and leak sensitive
information. This failure is particularly severe against indirect attacks,
highlighting a critical gap in current LLM safety alignment for sensitive
applications. Our analysis reveals that while models can often identify the
correct answer to a query, they struggle to incorporate policy constraints
during generation. In contrast, they exhibit a partial ability to revise
outputs when explicitly prompted. Our findings underscore the urgent need for
more robust methods to guarantee contextual security.
682eeb06720821973d643576
Hwan Chang
claimed_verified
2025-05-26T08:18:42.752Z
647c4a2692182942d7c2e698
/avatars/bcddf5fe49aa092a2645f70812108348.svg
HWANCHANG
user
HwanChang0106
682eeb06720821973d643577
Yumin Kim
claimed_verified
2025-05-26T14:07:44.617Z
64aac6984135aae75f3b99c1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yVR88SbW9TlJIrRHlJby-.jpeg
Yumin Kim
user
YuminKim
682eeb06720821973d643578
Yonghyun Jun
682eeb06720821973d643579
Hwanhee Lee
Keep Security! Benchmarking Security Policy Preservation in Large
  Language Model Contexts Against Indirect Attacks in Question Answering
As Large Language Models (LLMs) are increasingly deployed in sensitive
domains such as enterprise and government, ensuring that they adhere to
user-defined security policies within context is critical-especially with
respect to information non-disclosure. While prior LLM studies have focused on
general safety and socially sensitive data, large-scale benchmarks for
contextual security preservation against attacks remain lacking. To address
this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating
LLM adherence to contextual non-disclosure policies in question answering.
Derived from realistic contexts, our dataset includes explicit policies and
queries designed as direct and challenging indirect attacks seeking prohibited
information. We evaluate 10 LLMs on our benchmark and reveal a significant
vulnerability: many models violate user-defined policies and leak sensitive
information. This failure is particularly severe against indirect attacks,
highlighting a critical gap in current LLM safety alignment for sensitive
applications. Our analysis reveals that while models can often identify the
correct answer to a query, they struggle to incorporate policy constraints
during generation. In contrast, they exhibit a partial ability to revise
outputs when explicitly prompted. Our findings underscore the urgent need for
more robust methods to guarantee contextual security.
LLMs frequently violate contextual security policies by leaking sensitive information, particularly under indirect attacks, indicating a critical gap in current safety mechanisms.
Large Language Models (LLMs) security policies information non-disclosure CoPriva contextual security preservation question answering explicit policies indirect attacks virus policy constraints output revision