title: Tuna: Instruction Tuning using Feedback from Large Language Models
summary: Instruction tuning of open-source large language models (LLMs) like LLaMA,
  using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
  has proven to be a cost-effective way to align model behaviors with human
  preferences. However, the instruction-tuned model has only seen one response
  per instruction, lacking the knowledge of potentially better responses. In this
  paper, we propose finetuning an instruction-tuned LLM using our novel
  probabilistic ranking and contextual ranking approaches to
  increase the likelihood of generating better responses. Probabilistic ranking
  enables the instruction-tuned model to inherit the relative rankings of
  high-quality and low-quality responses from the teacher LLM. On the other hand,
  learning with contextual ranking allows the model to refine its own response
  distribution using the contextual understanding ability of stronger LLMs.
  Furthermore, we apply probabilistic ranking and contextual ranking sequentially
  to the instruction-tuned LLM. The resulting model, which we call Tuna,
  consistently improves the performance on Super Natural Instructions (119 test
  tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
  than several strong reinforcement learning baselines. Our code and data are
  available at  https://github.com/microsoft/LMOps.
paper_authors._id: 6535edac0917932040c4c475
paper_authors.name: Haoran Li
paper_authors._id: 6535edac0917932040c4c476
paper_authors.name: Yiran Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:47:42.148Z
paper_authors._id: 64ba2a5962c0bfc5739b0661
paper_authors.avatarUrl: /avatars/6c7b8495af6e7252de36b6e4071ca0a1.svg
paper_authors.fullname: Yiran Liu
paper_authors.type: user
paper_authors.user: liu-yr21
paper_authors._id: 6535edac0917932040c4c477
paper_authors.name: Xingxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:47:51.466Z
paper_authors._id: 64abbcff6cadc7aca584f71b
paper_authors.avatarUrl: /avatars/fc6e85ad4a8befd133a37b411712c648.svg
paper_authors.fullname: Xingxing Zhang
paper_authors.type: user
paper_authors.user: THU-CHUNXIA
paper_authors._id: 6535edac0917932040c4c478
paper_authors.name: Wei Lu
paper_authors._id: 6535edac0917932040c4c479
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:48:13.294Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Tuna: Instruction Tuning using Feedback from Large Language Models
paper_summary: Instruction tuning of open-source large language models (LLMs) like LLaMA,
  using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
  has proven to be a cost-effective way to align model behaviors with human
  preferences. However, the instruction-tuned model has only seen one response
  per instruction, lacking the knowledge of potentially better responses. In this
  paper, we propose finetuning an instruction-tuned LLM using our novel
  probabilistic ranking and contextual ranking approaches to
  increase the likelihood of generating better responses. Probabilistic ranking
  enables the instruction-tuned model to inherit the relative rankings of
  high-quality and low-quality responses from the teacher LLM. On the other hand,
  learning with contextual ranking allows the model to refine its own response
  distribution using the contextual understanding ability of stronger LLMs.
  Furthermore, we apply probabilistic ranking and contextual ranking sequentially
  to the instruction-tuned LLM. The resulting model, which we call Tuna,
  consistently improves the performance on Super Natural Instructions (119 test
  tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
  than several strong reinforcement learning baselines. Our code and data are
  available at  https://github.com/microsoft/LMOps.
paper_ai_keywords: instruction-tuning open-source large language models (LLMs) LLaMA Prior LLMs Instruct-GPT GPT-4 human preferences finetuning probabilistic ranking contextual ranking teacher LLM contextual understanding Super Natural Instructions LMentry Vicuna QA reinforcement learning baselines