title: Response Tuning: Aligning Large Language Models without Instruction
summary: Instruction tuning-supervised fine-tuning using instruction-response pairs-is
  a foundational step in transitioning pre-trained Large Language Models (LLMs)
  into helpful and safe chat assistants. Our hypothesis is that establishing an
  adequate output space can enable such a transition given the capabilities
  inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),
  which eliminates the instruction-conditioning step in instruction tuning and
  solely focuses on response space supervision. Our experiments demonstrate that
  RT models, trained only using responses, can effectively respond to a wide
  range of instructions and exhibit helpfulness comparable to that of their
  instruction-tuned counterparts. Furthermore, we observe that controlling the
  training response distribution can significantly improve their user preference
  or elicit target behaviors such as refusing assistance for unsafe queries. Our
  findings illuminate the role of establishing an adequate output space in
  alignment, highlighting the potential of the extensive inherent capabilities of
  pre-trained LLMs.
paper_authors._id: 6707b25a40b1f23d30085d85
paper_authors.name: Seokhyun An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T10:59:05.265Z
paper_authors._id: 617017b7544a1c62b298b87d
paper_authors.avatarUrl: /avatars/a569d976eb26807bd126de63a6b808fd.svg
paper_authors.fullname: Seokhyun An
paper_authors.type: user
paper_authors.user: seokhyun
paper_authors._id: 6707b25a40b1f23d30085d86
paper_authors.name: Hyounghun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T14:03:08.497Z
paper_authors._id: 64d7162b3ca2924d6e8593e4
paper_authors.avatarUrl: /avatars/51b28e4652bbb9d35adbd62aece59cd0.svg
paper_authors.fullname: Hyounghun Kim
paper_authors.type: user
paper_authors.user: khh3323
paper_title: Response Tuning: Aligning Large Language Models without Instruction
paper_summary: Instruction tuning-supervised fine-tuning using instruction-response pairs-is
  a foundational step in transitioning pre-trained Large Language Models (LLMs)
  into helpful and safe chat assistants. Our hypothesis is that establishing an
  adequate output space can enable such a transition given the capabilities
  inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),
  which eliminates the instruction-conditioning step in instruction tuning and
  solely focuses on response space supervision. Our experiments demonstrate that
  RT models, trained only using responses, can effectively respond to a wide
  range of instructions and exhibit helpfulness comparable to that of their
  instruction-tuned counterparts. Furthermore, we observe that controlling the
  training response distribution can significantly improve their user preference
  or elicit target behaviors such as refusing assistance for unsafe queries. Our
  findings illuminate the role of establishing an adequate output space in
  alignment, highlighting the potential of the extensive inherent capabilities of
  pre-trained LLMs.
paper_ai_summary: Response Tuning, an approach focusing on response space supervision without instruction conditioning, effectively transitions pre-trained LLMs into helpful and safe chat assistants.
paper_ai_keywords: instruction tuning instruction-response pairs Large Language Models (LLMs) Response Tuning (RT) response space supervision user preference target behaviors alignment capabilities