Recovering from Privacy-Preserving Masking with Large Language Models
Model adaptation is crucial to handle the discrepancy between proxy training
data and actual users data received. To effectively perform adaptation, textual
data of users is typically stored on servers or their local devices, where
downstream natural language processing (NLP) models can be directly trained
using such in-domain data. However, this might raise privacy and security
concerns due to the extra risks of exposing user information to adversaries.
Replacing identifying information in textual data with a generic marker has
been recently explored. In this work, we leverage large language models (LLMs)
to suggest substitutes of masked tokens and have their effectiveness evaluated
on downstream language modeling tasks. Specifically, we propose multiple
pre-trained and fine-tuned LLM-based approaches and perform empirical studies
on various datasets for the comparison of these methods. Experimental results
show that models trained on the obfuscation corpora are able to achieve
comparable performance with the ones trained on the original data without
privacy-preserving token masking.
6509008610173b169c32060e
Arpita Vats
admin_assigned
2023-09-20T14:42:27.145Z
64c164225258457cc523de88
/avatars/e931f9fbbceff10cbb025f22aef0305b.svg
Arpita Vats
user
arpita08
6509008610173b169c32060f
Zhe Liu
6509008610173b169c320610
Peng Su
6509008610173b169c320611
Debjyoti Paul
6509008610173b169c320612
Yingyi Ma
admin_assigned
2023-09-20T14:43:51.698Z
64c7221e9cff58f22c584ad1
/avatars/a39b3f67a3a980b405dcc906458e8b4a.svg
yingyima
user
yingyima
6509008610173b169c320613
Yutong Pang
6509008610173b169c320614
Zeeshan Ahmed
6509008610173b169c320615
Ozlem Kalinli
Recovering from Privacy-Preserving Masking with Large Language Models
Model adaptation is crucial to handle the discrepancy between proxy training
data and actual users data received. To effectively perform adaptation, textual
data of users is typically stored on servers or their local devices, where
downstream natural language processing (NLP) models can be directly trained
using such in-domain data. However, this might raise privacy and security
concerns due to the extra risks of exposing user information to adversaries.
Replacing identifying information in textual data with a generic marker has
been recently explored. In this work, we leverage large language models (LLMs)
to suggest substitutes of masked tokens and have their effectiveness evaluated
on downstream language modeling tasks. Specifically, we propose multiple
pre-trained and fine-tuned LLM-based approaches and perform empirical studies
on various datasets for the comparison of these methods. Experimental results
show that models trained on the obfuscation corpora are able to achieve
comparable performance with the ones trained on the original data without
privacy-preserving token masking.
LLMs replace identifying information in textual data to achieve comparable performance in language modeling tasks while preserving privacy.
large language models LLMs pre-trained fine-tuned obfuscation corpora language modeling tasks privacy-preserving token masking