Toxicity. We use ToxiGen (Hartvigsen et al., 2022) to quantify the extent of toxic language and hate
speech generation across various demographic groups. The ToxiGen dataset contains implicitly toxic and
benign sentences mentioning 13 minority groups. Following Touvron et al. (2023b), we utilize an improved
version of the dataset, which minimizes noise by removing prompts with disagreements among annotators
regarding the target demographic group. To measure the toxicity of the generated outputs from each of the
LLMs, we employ the default ToxiGen classifier, tuned on RoBERTa (Liu et al., 2019).
CodeLlama2