ChatGPT. Figure 2 displays ChatGPT’s com-
mon responses to DP, JP and MJP. The case of
DP shows ChatGPT’s moral sense to value indi-

Figure 2: ChatGPT’s responses to various prompts.

Figure 3: The New Bing’s dialog case for DP.

viduals’ privacy. Its ethical modules are effective
against common prompts regarding personal infor-
mation. Moreover, as shown in the case of JP,
ChatGPT may sometimes refuse to answer such
queries under role-play based jailbreaking prompts.
However, ChatGPT may give unethical comments
like hacking databases under the “Developer Mode”
of jailbreaking prompts. For MJP, ChatGPT is
more willing to generate personal information if
we ask it to make random guesses. Regrettably,
some random guesses may be correct. These re-
sults imply that ChatGPT fails to defend against
indirect and vicious prompts and more defenses on
the dialog-safety should be employed.
Multi-step Jailbreaking Privacy Attacks on ChatGPT