Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models
<|reference_start|>Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models: Large Language Models (LLMs) exhibit impressive capabilities but also present risks such as biased content generation and privacy issues. One of the current alignment techniques includes principle-driven integration, but it faces challenges arising from the imprecision of manually crafted rules and inadequate risk perception in models without safety training. To address these, we introduce Guide-Align, a two-stage approach. Initially, a safety-trained model identifies potential risks and formulates specific guidelines for various inputs, establishing a comprehensive library of guidelines and a model for input-guidelines retrieval. Subsequently, the retrieval model correlates new inputs with relevant guidelines, which guide LLMs in response generation to ensure safe and high-quality outputs, thereby aligning with human values. An additional optional stage involves fine-tuning a model with well-aligned datasets generated through the process implemented in the second stage. Our method customizes guidelines to accommodate diverse inputs, thereby enhancing the fine-grainedness and comprehensiveness of the guideline library. Furthermore, it incorporates safety expertise from a safety-trained LLM through a lightweight retrieval model. We evaluate our approach on three benchmarks, demonstrating significant improvements in LLM security and quality. Notably, our fine-tuned model, Labrador, even at 13 billion parameters, outperforms GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.<|reference_end|>
@article{luo2024ensuring,
    title={Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for
  Language Models},
    author={Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu,
  Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong},
    journal={arXiv preprint arXiv:2403.11838},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11838},
    primaryClass={cs.CL cs.AI}
}