3.3.2 Extraction with Jailbreaking Prompts
Though ChatGPT pays great effort into dialog
safety and can successfully prevent against train-
ing data extraction attacks with direct prompts,
there is still a sideway to bypass ChatGPT’s ethi-
cal modules called jailbreaking. Jailbreaking ex-
ploits tricky prompts to make ChatGPT evade pro-
gramming restrictions and generate anything freely.
These tricky prompts usually set up user-created
role plays to alter ChatGPT’s ego and allow Chat-
GPT to answer user queries unethically. DAN
refers to “Do Anything for Now”, and is one exem-
plary jailbreaking prompt to generate offensive or
prejudiced comments about politics, race and sex.
In this work, we exploit these jailbreaking
prompts to make ChatGPT generate personal infor-
mation of given names. For example, according to
the use cases of Figure 1 (b), ChatGPT sometimes
generates private information from its “Developer
Mode” role of the jailbreaking prompt.
Multi-step Jailbreaking Privacy Attacks on ChatGPT