One-Shot Safety Alignment for Large Language Models via Optimal Dualization
<|reference_start|>One-Shot Safety Alignment for Large Language Models via Optimal Dualization: The growing safety concerns surrounding Large Language Models (LLMs) raise an urgent need to align them with diverse human preferences to simultaneously enhance their helpfulness and safety. A promising approach is to enforce safety constraints through Reinforcement Learning from Human Feedback (RLHF). For such constrained RLHF, common Lagrangian-based primal-dual policy optimization methods are computationally expensive and often unstable. This paper presents a dualization perspective that reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. This shortcut eliminates the need for cumbersome primal-dual policy iterations, thus greatly reducing the computational burden and improving training stability. Our strategy leads to two practical algorithms in model-based and preference-based scenarios (MoCAN and PeCAN, respectively). A broad range of experiments demonstrate the effectiveness of our methods.<|reference_end|>
@article{huang2024one-shot,
    title={One-Shot Safety Alignment for Large Language Models via Optimal
  Dualization},
    author={Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani,
  Dongsheng Ding},
    journal={arXiv preprint arXiv:2405.19544},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19544},
    primaryClass={cs.AI cs.CL cs.LG math.OC stat.ML}
}