title: PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
    Action
summary: As language models (LMs) are widely utilized in personalized communication
  scenarios (e.g., sending emails, writing social media posts) and endowed with a
  certain level of agency, ensuring they act in accordance with the contextual
  privacy norms becomes increasingly critical. However, quantifying the privacy
  norm awareness of LMs and the emerging privacy risk in LM-mediated
  communication is challenging due to (1) the contextual and long-tailed nature
  of privacy-sensitive cases, and (2) the lack of evaluation approaches that
  capture realistic application scenarios. To address these challenges, we
  propose PrivacyLens, a novel framework designed to extend privacy-sensitive
  seeds into expressive vignettes and further into agent trajectories, enabling
  multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
  PrivacyLens with a collection of privacy norms grounded in privacy literature
  and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
  performance in answering probing questions and their actual behavior when
  executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
  and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
  when prompted with privacy-enhancing instructions. We also demonstrate the
  dynamic nature of PrivacyLens by extending each seed into multiple trajectories
  to red-team LM privacy leakage risk. Dataset and code are available at
  https://github.com/SALT-NLP/PrivacyLens.
paper_authors._id: 66d8907f6c68f386ac2b571a
paper_authors.name: Yijia Shao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-04T18:30:00.989Z
paper_authors._id: 6171514808b4ddb7392f2eb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
paper_authors.fullname: Yijia Shao
paper_authors.type: user
paper_authors.user: EchoShao8899
paper_authors._id: 66d8907f6c68f386ac2b571b
paper_authors.name: Tianshi Li
paper_authors._id: 66d8907f6c68f386ac2b571c
paper_authors.name: Weiyan Shi
paper_authors._id: 66d8907f6c68f386ac2b571d
paper_authors.name: Yanchen Liu
paper_authors._id: 66d8907f6c68f386ac2b571e
paper_authors.name: Diyi Yang
paper_title: PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
    Action
paper_summary: As language models (LMs) are widely utilized in personalized communication
  scenarios (e.g., sending emails, writing social media posts) and endowed with a
  certain level of agency, ensuring they act in accordance with the contextual
  privacy norms becomes increasingly critical. However, quantifying the privacy
  norm awareness of LMs and the emerging privacy risk in LM-mediated
  communication is challenging due to (1) the contextual and long-tailed nature
  of privacy-sensitive cases, and (2) the lack of evaluation approaches that
  capture realistic application scenarios. To address these challenges, we
  propose PrivacyLens, a novel framework designed to extend privacy-sensitive
  seeds into expressive vignettes and further into agent trajectories, enabling
  multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
  PrivacyLens with a collection of privacy norms grounded in privacy literature
  and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
  performance in answering probing questions and their actual behavior when
  executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
  and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
  when prompted with privacy-enhancing instructions. We also demonstrate the
  dynamic nature of PrivacyLens by extending each seed into multiple trajectories
  to red-team LM privacy leakage risk. Dataset and code are available at
  https://github.com/SALT-NLP/PrivacyLens.
paper_ai_keywords: PrivacyLens privacy-sensitive seeds expressive vignettes agent trajectories privacy leakage privacy-sensitive cases LM agents probing questions privacy-enhancing instructions red-team LM privacy leakage risk