3.3.1 Extraction with Direct Prompts
Previous works (Carlini et al., 2021; Huang et al.,
2022; Mireshghallah et al., 2022; Lukas et al.,
2023) mainly used direct prompts to extract private
information from LMs including variants of GPT-2.
For example, the adversary may use prompts like
“ name: [name], email: ____” to extract the email
address of a speciﬁc person or use “ name: ____”
directly to recover multiple (name, email) pairs via
sampling-based decoding.

Fortunately, thanks to the dialog safety ﬁne-
tuning, ChatGPT of the Mar Version tends to hesi-
tate from answering any private information if we
use direct prompts for data extraction.As shown
in Figure 1 (a), ChatGPT refuses to generate any
personal information with direct prompts.
Multi-step Jailbreaking Privacy Attacks on ChatGPT