Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models
<|reference_start|>Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models: Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.<|reference_end|>
@article{wu2024dancing,
    title={Dancing in Chains: Reconciling Instruction Following and Faithfulness in
  Language Models},
    author={Zhengxuan Wu and Yuhao Zhang and Peng Qi and Yumo Xu and Rujun Han and
  Yian Zhang and Jifan Chen and Bonan Min and Zhiheng Huang},
    journal={arXiv preprint arXiv:2407.21417},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.21417},
    primaryClass={cs.CL}
}