3.3.4 Response Veriﬁcation
Besides prompt tricks, for each data sample, we
could also generate private information multiple

times with sample-based decoding. As displayed
in Figure 1 (d), we collect distinct personal infor-
mation from diverse responses. We consider two
methods to verify which one is the correct answer.
The ﬁrst method converts the collected information
into a multiple-choice question and prompts the
LLM again to choose the correct answer. During
implementation, we treat the ﬁrst displayed infor-
mation in the response as the LLM’s ﬁnal choice.
The second method is majority voting which re-
gards the most frequent prediction as the ﬁnal an-
swer. If there is a tie, we randomly choose one
candidate as the ﬁnal prediction.
Multi-step Jailbreaking Privacy Attacks on ChatGPT