title: How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
    Resources
summary: In this work we explore recent advances in instruction-tuning language models
  on a range of open instruction-following datasets. Despite recent claims that
  open models can be on par with state-of-the-art proprietary models, these
  claims are often accompanied by limited evaluation, making it difficult to
  compare models across the board and determine the utility of various resources.
  We provide a large set of instruction-tuned models from 6.7B to 65B parameters
  in size, trained on 12 instruction datasets ranging from manually curated
  (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
  systematically evaluate them on their factual knowledge, reasoning,
  multilinguality, coding, and open-ended instruction following abilities through
  a collection of automatic, model-based, and human-based metrics. We further
  introduce T\"ulu, our best performing instruction-tuned model suite finetuned
  on a combination of high-quality open resources.
    Our experiments show that different instruction-tuning datasets can uncover
  or enhance specific skills, while no single dataset (or combination) provides
  the best performance across all evaluations. Interestingly, we find that model
  and human preference-based evaluations fail to reflect differences in model
  capabilities exposed by benchmark-based evaluations, suggesting the need for
  the type of systemic evaluation performed in this work. Our evaluations show
  that the best model in any given evaluation reaches on average 83% of ChatGPT
  performance, and 68% of GPT-4 performance, suggesting that further investment
  in building better base models and instruction-tuning data is required to close
  the gap. We release our instruction-tuned models, including a fully finetuned
  65B T\"ulu, along with our code, data, and evaluation framework at
  https://github.com/allenai/open-instruct to facilitate future research.
paper_authors._id: 648270672bc908e2d9e2a0d4
paper_authors.name: Yizhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:18:21.440Z
paper_authors._id: 6269be67d1ac0cde592aba29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
paper_authors.fullname: Yizhong Wang
paper_authors.type: user
paper_authors.user: yizhongw
paper_authors._id: 648270672bc908e2d9e2a0d5
paper_authors.name: Hamish Ivison
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:32:10.401Z
paper_authors._id: 62608fc2ffe8827cb1d89f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
paper_authors.fullname: Hamish Ivison
paper_authors.type: user
paper_authors.user: hamishivi
paper_authors._id: 648270672bc908e2d9e2a0d6
paper_authors.name: Pradeep Dasigi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:18:41.250Z
paper_authors._id: 6408fcc93461c51cf735a61e
paper_authors.avatarUrl: /avatars/619f3653911d111f046a5a6c30fc8319.svg
paper_authors.fullname: Pradeep Dasigi
paper_authors.type: user
paper_authors.user: pradeepd
paper_authors._id: 648270672bc908e2d9e2a0d7
paper_authors.name: Jack Hessel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:19:08.476Z
paper_authors._id: 625498644f4edf771516b2cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
paper_authors.fullname: Jack Hessel
paper_authors.type: user
paper_authors.user: jmhessel
paper_authors._id: 648270672bc908e2d9e2a0d8
paper_authors.name: Tushar Khot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:19:27.010Z
paper_authors._id: 62cdf2dbaac2c91c95581830
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
paper_authors.fullname: Tushar Khot
paper_authors.type: user
paper_authors.user: tusharkhot
paper_authors._id: 648270672bc908e2d9e2a0d9
paper_authors.name: Khyathi Raghavi Chandu
paper_authors._id: 648270672bc908e2d9e2a0da
paper_authors.name: David Wadden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:19:55.961Z
paper_authors._id: 5ff4e2a1463be69ae4bd42bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
paper_authors.fullname: David Wadden
paper_authors.type: user
paper_authors.user: dwadden
paper_authors._id: 648270672bc908e2d9e2a0db
paper_authors.name: Kelsey MacMillan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:20:21.497Z
paper_authors._id: 64528954c895e6437dc4be98
paper_authors.avatarUrl: /avatars/685e3f4bf768a372a30615ec37b0f25b.svg
paper_authors.fullname: Kelsey MacMillan
paper_authors.type: user
paper_authors.user: kmacmillan
paper_authors._id: 648270672bc908e2d9e2a0dc
paper_authors.name: Noah A. Smith
paper_authors._id: 648270672bc908e2d9e2a0dd
paper_authors.name: Iz Beltagy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:20:45.755Z
paper_authors._id: 5ec07694ed25d76864d553f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
paper_authors.fullname: Iz Beltagy
paper_authors.type: user
paper_authors.user: beltagy
paper_authors._id: 648270672bc908e2d9e2a0de
paper_authors.name: Hannaneh Hajishirzi
paper_title: How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
    Resources
paper_summary: In this work we explore recent advances in instruction-tuning language models
  on a range of open instruction-following datasets. Despite recent claims that
  open models can be on par with state-of-the-art proprietary models, these
  claims are often accompanied by limited evaluation, making it difficult to
  compare models across the board and determine the utility of various resources.
  We provide a large set of instruction-tuned models from 6.7B to 65B parameters
  in size, trained on 12 instruction datasets ranging from manually curated
  (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
  systematically evaluate them on their factual knowledge, reasoning,
  multilinguality, coding, and open-ended instruction following abilities through
  a collection of automatic, model-based, and human-based metrics. We further
  introduce T\"ulu, our best performing instruction-tuned model suite finetuned
  on a combination of high-quality open resources.
    Our experiments show that different instruction-tuning datasets can uncover
  or enhance specific skills, while no single dataset (or combination) provides
  the best performance across all evaluations. Interestingly, we find that model
  and human preference-based evaluations fail to reflect differences in model
  capabilities exposed by benchmark-based evaluations, suggesting the need for
  the type of systemic evaluation performed in this work. Our evaluations show
  that the best model in any given evaluation reaches on average 83% of ChatGPT
  performance, and 68% of GPT-4 performance, suggesting that further investment
  in building better base models and instruction-tuning data is required to close
  the gap. We release our instruction-tuned models, including a fully finetuned
  65B T\"ulu, along with our code, data, and evaluation framework at
  https://github.com/allenai/open-instruct to facilitate future research.
paper_ai_keywords: instruction-tuning language models instruction-following datasets open models proprietary models instruction-tuned models parameters OpenAssistant Alpaca instruction-following abilities factual knowledge reasoning multilinguality coding automatic metrics model-based metrics human-based metrics T\"ulu benchmark-based evaluations ChatGPT GPT-4