Preference-Guided Reflective Sampling for Aligning Language Models
<|reference_start|>Preference-Guided Reflective Sampling for Aligning Language Models: Iterative data generation and model re-training can effectively align large language models(LLMs) to human preferences. The process of data sampling is crucial, as it significantly influences the success of policy improvement. Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs. In this work, we propose a more effective sampling method, named Preference-Guided Reflective Sampling (PRS). Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. It leverages adaptive self-refinement techniques to better explore the sampling space. By specifying user preferences in natural language, PRS can further optimize response generation according to these preferences. As a result, PRS can align models to diverse user preferences. Our experiments demonstrate that PRS generates higher-quality responses with significantly higher rewards. On AlpacaEval and Arena-Hard, PRS substantially outperforms repeated random sampling in best-of-$N$ sampling. Moreover, PRS shows strong performance when applied in iterative offline RL training.<|reference_end|>
@article{ye2024preference-guided,
    title={Preference-Guided Reflective Sampling for Aligning Language Models},
    author={Hai Ye, Hwee Tou Ng},
    journal={arXiv preprint arXiv:2408.12163},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.12163},
    primaryClass={cs.CL}
}