trained simply by language modeling objective,
which aimed to predict the next word given the cur-
rent context. Despite the efforts made by these pio-
neering works, there is still a huge gap between the
latest LLMs and GPT-2. First, LLMs’ model sizes
and dataset scales are much larger than GPT-2. Sec-
ond, LLMs implement more sophisticated training
objectives, which include instruction tuning (Wei
et al., 2022a) and Reinforcement Learning from
Human Feedback (RLHF) (Christiano et al., 2017).
Third, most LLMs only provide application pro-
gramming interfaces (APIs) and we cannot inspect
the model weights and training corpora. Lastly, it
is trending to integrate various applications into
LLMs to empower LLMs’ knowledge grounding
ability to solve math problems (ChatGPT + Wol-
fram Alpha), read formatted ﬁles (ChatPDF), and
respond to queries with the search engine (the New
Bing). As a result, it remains unknown to what
extent privacy leakage occurs on these present-day
Multi-step Jailbreaking Privacy Attacks on ChatGPT