PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
  Action
As language models (LMs) are widely utilized in personalized communication
scenarios (e.g., sending emails, writing social media posts) and endowed with a
certain level of agency, ensuring they act in accordance with the contextual
privacy norms becomes increasingly critical. However, quantifying the privacy
norm awareness of LMs and the emerging privacy risk in LM-mediated
communication is challenging due to (1) the contextual and long-tailed nature
of privacy-sensitive cases, and (2) the lack of evaluation approaches that
capture realistic application scenarios. To address these challenges, we
propose PrivacyLens, a novel framework designed to extend privacy-sensitive
seeds into expressive vignettes and further into agent trajectories, enabling
multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
PrivacyLens with a collection of privacy norms grounded in privacy literature
and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
performance in answering probing questions and their actual behavior when
executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
when prompted with privacy-enhancing instructions. We also demonstrate the
dynamic nature of PrivacyLens by extending each seed into multiple trajectories
to red-team LM privacy leakage risk. Dataset and code are available at
https://github.com/SALT-NLP/PrivacyLens.
66d8907f6c68f386ac2b571a
Yijia Shao
extracted_confirmed
2024-09-04T18:30:00.989Z
6171514808b4ddb7392f2eb0
https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
Yijia Shao
user
EchoShao8899
66d8907f6c68f386ac2b571b
Tianshi Li
66d8907f6c68f386ac2b571c
Weiyan Shi
66d8907f6c68f386ac2b571d
Yanchen Liu
66d8907f6c68f386ac2b571e
Diyi Yang
PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
  Action
As language models (LMs) are widely utilized in personalized communication
scenarios (e.g., sending emails, writing social media posts) and endowed with a
certain level of agency, ensuring they act in accordance with the contextual
privacy norms becomes increasingly critical. However, quantifying the privacy
norm awareness of LMs and the emerging privacy risk in LM-mediated
communication is challenging due to (1) the contextual and long-tailed nature
of privacy-sensitive cases, and (2) the lack of evaluation approaches that
capture realistic application scenarios. To address these challenges, we
propose PrivacyLens, a novel framework designed to extend privacy-sensitive
seeds into expressive vignettes and further into agent trajectories, enabling
multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
PrivacyLens with a collection of privacy norms grounded in privacy literature
and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
performance in answering probing questions and their actual behavior when
executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
when prompted with privacy-enhancing instructions. We also demonstrate the
dynamic nature of PrivacyLens by extending each seed into multiple trajectories
to red-team LM privacy leakage risk. Dataset and code are available at
https://github.com/SALT-NLP/PrivacyLens.
PrivacyLens evaluates privacy leakage in language model agents by extending privacy seeds into scenarios, revealing discrepancies between responses to probing questions and actual user instruction execution.
language models privacy norms privacy-sensitive vignettes agent trajectories privacy leakage probing questions privacy-enhancing instructions red-team GPT-4 Llama-3-70B