less attention to the data quality and conﬁdential-
ity of the web-sourced data (Piktus et al., 2023).
Such mass collection of personal data incurs de-
bates and worries. For example, under the EU’s
General Data Protection Regulation (GDPR), train-
ing a commercial model on extensive personal data
without notice or consent from data subjects lacks
a legal basis. Consequently, Italy once temporarily
banned ChatGPT due to privacy considerations1.
Unfortunately, the privacy analysis of language
models is still less explored and remains an active
area. Prior works (Lukas et al., 2023; Pan et al.,
2020; Mireshghallah et al., 2022; Huang et al.,
2022; Carlini et al., 2021) studied the privacy leak-
age issues of language models (LMs) and claimed
that memorizing training data leads to private data
leakage. However, these works mainly investigated
variants of GPT-2 models (Radford et al., 2019)
trained simply by language modeling objective,
which aimed to predict the next word given the cur-
Multi-step Jailbreaking Privacy Attacks on ChatGPT