title: Recovering from Privacy-Preserving Masking with Large Language Models
summary: Model adaptation is crucial to handle the discrepancy between proxy training
  data and actual users data received. To effectively perform adaptation, textual
  data of users is typically stored on servers or their local devices, where
  downstream natural language processing (NLP) models can be directly trained
  using such in-domain data. However, this might raise privacy and security
  concerns due to the extra risks of exposing user information to adversaries.
  Replacing identifying information in textual data with a generic marker has
  been recently explored. In this work, we leverage large language models (LLMs)
  to suggest substitutes of masked tokens and have their effectiveness evaluated
  on downstream language modeling tasks. Specifically, we propose multiple
  pre-trained and fine-tuned LLM-based approaches and perform empirical studies
  on various datasets for the comparison of these methods. Experimental results
  show that models trained on the obfuscation corpora are able to achieve
  comparable performance with the ones trained on the original data without
  privacy-preserving token masking.
paper_authors._id: 6509008610173b169c32060e
paper_authors.name: Arpita Vats
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:42:27.145Z
paper_authors._id: 64c164225258457cc523de88
paper_authors.avatarUrl: /avatars/e931f9fbbceff10cbb025f22aef0305b.svg
paper_authors.fullname: Arpita Vats
paper_authors.type: user
paper_authors.user: arpita08
paper_authors._id: 6509008610173b169c32060f
paper_authors.name: Zhe Liu
paper_authors._id: 6509008610173b169c320610
paper_authors.name: Peng Su
paper_authors._id: 6509008610173b169c320611
paper_authors.name: Debjyoti Paul
paper_authors._id: 6509008610173b169c320612
paper_authors.name: Yingyi Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:43:51.698Z
paper_authors._id: 64c7221e9cff58f22c584ad1
paper_authors.avatarUrl: /avatars/a39b3f67a3a980b405dcc906458e8b4a.svg
paper_authors.fullname: yingyima
paper_authors.type: user
paper_authors.user: yingyima
paper_authors._id: 6509008610173b169c320613
paper_authors.name: Yutong Pang
paper_authors._id: 6509008610173b169c320614
paper_authors.name: Zeeshan Ahmed
paper_authors._id: 6509008610173b169c320615
paper_authors.name: Ozlem Kalinli
paper_title: Recovering from Privacy-Preserving Masking with Large Language Models
paper_summary: Model adaptation is crucial to handle the discrepancy between proxy training
  data and actual users data received. To effectively perform adaptation, textual
  data of users is typically stored on servers or their local devices, where
  downstream natural language processing (NLP) models can be directly trained
  using such in-domain data. However, this might raise privacy and security
  concerns due to the extra risks of exposing user information to adversaries.
  Replacing identifying information in textual data with a generic marker has
  been recently explored. In this work, we leverage large language models (LLMs)
  to suggest substitutes of masked tokens and have their effectiveness evaluated
  on downstream language modeling tasks. Specifically, we propose multiple
  pre-trained and fine-tuned LLM-based approaches and perform empirical studies
  on various datasets for the comparison of these methods. Experimental results
  show that models trained on the obfuscation corpora are able to achieve
  comparable performance with the ones trained on the original data without
  privacy-preserving token masking.
paper_ai_summary: LLMs replace identifying information in textual data to achieve comparable performance in language modeling tasks while preserving privacy.
paper_ai_keywords: large language models LLMs pre-trained fine-tuned obfuscation corpora language modeling tasks privacy-preserving token masking