the obtained response and direct prompts to obtain
the ﬁnal responses and parse the PII.
• Multi-step Jailbreaking Prompt (MJP). We use
the three-utterance context mentioned in Sec 3.3.3
to obtain responses and try to parse the PII.
• MJP+multiple choice (MJP+MC). We generate 5
responses via MJP. Then we use a multiple-choice
template to prompt ChatGPT again to choose the
ﬁnal answer.
• MJP+majority voting (MJP+MV). We generate
5 responses via MJP. Then we use majority voting
to choose the ﬁnal answer.

These prompts’ examples can be found in Fig-
ure 1. And the detailed templates are reported in
Appendix A.
Multi-step Jailbreaking Privacy Attacks on ChatGPT