Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
As the era of autonomous agents making decisions on behalf of users unfolds,
ensuring contextual integrity (CI) -- what is the appropriate information to
share while carrying out a certain task -- becomes a central question to the
field. We posit that CI demands a form of reasoning where the agent needs to
reason about the context in which it is operating. To test this, we first
prompt LLMs to reason explicitly about CI when deciding what information to
disclose. We then extend this approach by developing a reinforcement learning
(RL) framework that further instills in models the reasoning necessary to
achieve CI. Using a synthetic, automatically created, dataset of only sim700
examples but with diverse contexts and information disclosure norms, we show
that our method substantially reduces inappropriate information disclosure
while maintaining task performance across multiple model sizes and families.
Importantly, improvements transfer from this synthetic dataset to established
CI benchmarks such as PrivacyLens that has human annotations and evaluates
privacy leakage of AI assistants in actions and tool calls.
68425054feb46a093178003f
Guangchen Lan
extracted_confirmed
2025-06-06T02:20:41.949Z
64ff4b1a0e8369f6a8c47c7e
https://cdn-avatars.huggingface.co/v1/production/uploads/64ff4b1a0e8369f6a8c47c7e/X8ocOagj89gexSa253q_8.png
Eric Lan
user
Eric-Lan
68425054feb46a0931780040
Huseyin A. Inan
68425054feb46a0931780041
Sahar Abdelnabi
extracted_confirmed
2025-06-06T10:29:16.120Z
65e88cdd95a27dfbf6b4e63b
/avatars/3d2d270398f0824b392f99e158e94f26.svg
Sahar Abdelnabi
user
sahar-abdelnabi
68425054feb46a0931780042
Janardhan Kulkarni
68425054feb46a0931780043
Lukas Wutschitz
extracted_confirmed
2025-06-06T09:45:54.243Z
6380a37a5c62156ce7dff8b9
/avatars/fbe5a20869cb55ec43759c1b5f9c4135.svg
Lukas Wutschitz
user
wulu
68425054feb46a0931780044
Reza Shokri
68425054feb46a0931780045
Christopher G. Brinton
68425054feb46a0931780046
Robert Sim
Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
As the era of autonomous agents making decisions on behalf of users unfolds,
ensuring contextual integrity (CI) -- what is the appropriate information to
share while carrying out a certain task -- becomes a central question to the
field. We posit that CI demands a form of reasoning where the agent needs to
reason about the context in which it is operating. To test this, we first
prompt LLMs to reason explicitly about CI when deciding what information to
disclose. We then extend this approach by developing a reinforcement learning
(RL) framework that further instills in models the reasoning necessary to
achieve CI. Using a synthetic, automatically created, dataset of only sim700
examples but with diverse contexts and information disclosure norms, we show
that our method substantially reduces inappropriate information disclosure
while maintaining task performance across multiple model sizes and families.
Importantly, improvements transfer from this synthetic dataset to established
CI benchmarks such as PrivacyLens that has human annotations and evaluates
privacy leakage of AI assistants in actions and tool calls.
A reinforcement learning framework for LLMs enhances contextual integrity by reducing inappropriate information disclosure and maintaining task performance across various benchmarks.
LLMs reinforcement learning contextual integrity information disclosure synthetic dataset PrivacyLens privacy leakage AI assistants