for open-source LLMs. To democratize endeavors on advancing the safety alignment of LLMs, Ji
et al. (2023) gathers a human-preference dataset to disentangle harmlessness and helpfulness from
the human-preference score, thus providing separate ranking data for the two metrics. Experiments
show that disentangling human preferences enhances safety alignment. Bai et al. (2022b) seeks to
increase safety with RL from AI Feedback (RLAIF), where the preference model is trained using
LLM-generated self-critiques and revisions. Direct Preference Optimization (DPO) (Rafailov et al.,
2023a) reduces the need to learn a reward model and learn from preferences directly with a simple
cross-entropy loss, which could largely reduce costs for RLHF. Combining and improving these
methodologies could lead to potential improvements in safety for open-source LLMs.

4 Discussion

4.1 Development Trend of LLMs
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup