III. EXPERIMENTS

A. Victim model and Dataset

The victim model consists of a PLM model and a re-
ward model. Following the work of [2], we use GPT-2 and
DistillBert as benchmarks for our initial experiments. GPT-2
is a large language model based on transformer architecture
and contains 1.5 billion parameters. DistillBert, on the other
hand, is a miniaturized version of Bert that is trained using
knowledge distillation technology. In our experiments, GPT-
2 serves as the PLM while DistillBert serves as the reward
model. We use IMDB, a sentiment analysis dataset containing
50,000 pairs of data, as the feedback for human preferences.

B. Attack Results
BadGPT- Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT