title: Stronger Models are NOT Stronger Teachers for Instruction Tuning
summary: Instruction tuning has been widely adopted to ensure large language models
  (LLMs) follow user instructions effectively. The resulting
  instruction-following capabilities of LLMs heavily rely on the instruction
  datasets used for tuning. Recently, synthetic instruction datasets have emerged
  as an economically viable solution to provide LLMs diverse and high-quality
  instructions. However, existing approaches typically assume that larger or
  stronger models are stronger teachers for instruction tuning, and hence simply
  adopt these models as response generators to the synthetic instructions. In
  this paper, we challenge this commonly-adopted assumption. Our extensive
  experiments across five base models and twenty response generators reveal that
  larger and stronger models are not necessarily stronger teachers of smaller
  models. We refer to this phenomenon as the Larger Models' Paradox. We observe
  that existing metrics cannot precisely predict the effectiveness of response
  generators since they ignore the compatibility between teachers and base models
  being fine-tuned. We thus develop a novel metric, named as
  Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
  generators. Our experiments across five base models demonstrate that CAR
  outperforms almost all baselines.
paper_authors._id: 6734679ad3521b36246eb784
paper_authors.name: Zhangchen Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:24:56.226Z
paper_authors._id: 653df1323479e9ebbe3eb6cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
paper_authors.fullname: Zhangchen Xu
paper_authors.type: user
paper_authors.user: zhangchenxu
paper_authors._id: 6734679ad3521b36246eb785
paper_authors.name: Fengqing Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:25:03.632Z
paper_authors._id: 6531e1021dd8ebbdc1a6fd8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6531e1021dd8ebbdc1a6fd8e/lIcl7zCPtzRsfiUh6uY1o.jpeg
paper_authors.fullname: Fengqing Jiang
paper_authors.type: user
paper_authors.user: fqjiang
paper_authors._id: 6734679ad3521b36246eb786
paper_authors.name: Luyao Niu
paper_authors._id: 6734679ad3521b36246eb787
paper_authors.name: Bill Yuchen Lin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-13T08:47:23.435Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 6734679ad3521b36246eb788
paper_authors.name: Radha Poovendran
paper_title: Stronger Models are NOT Stronger Teachers for Instruction Tuning
paper_summary: Instruction tuning has been widely adopted to ensure large language models
  (LLMs) follow user instructions effectively. The resulting
  instruction-following capabilities of LLMs heavily rely on the instruction
  datasets used for tuning. Recently, synthetic instruction datasets have emerged
  as an economically viable solution to provide LLMs diverse and high-quality
  instructions. However, existing approaches typically assume that larger or
  stronger models are stronger teachers for instruction tuning, and hence simply
  adopt these models as response generators to the synthetic instructions. In
  this paper, we challenge this commonly-adopted assumption. Our extensive
  experiments across five base models and twenty response generators reveal that
  larger and stronger models are not necessarily stronger teachers of smaller
  models. We refer to this phenomenon as the Larger Models' Paradox. We observe
  that existing metrics cannot precisely predict the effectiveness of response
  generators since they ignore the compatibility between teachers and base models
  being fine-tuned. We thus develop a novel metric, named as
  Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
  generators. Our experiments across five base models demonstrate that CAR
  outperforms almost all baselines.
paper_ai_keywords: instruction tuning large language models (LLMs) instruction-following capabilities instruction datasets synthetic instruction datasets response generators Larger Models' Paradox Compatibility-Adjusted Reward (CAR)