title: INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
    Language Models
summary: Instruction-tuned large language models have revolutionized natural language
  processing and have shown great potential in applications such as
  conversational agents. These models, such as GPT-4, can not only master
  language but also solve complex tasks in areas like mathematics, coding,
  medicine, and law. Despite their impressive capabilities, there is still a lack
  of comprehensive understanding regarding their full potential, primarily due to
  the black-box nature of many models and the absence of holistic evaluation
  studies. To address these challenges, we present INSTRUCTEVAL, a more
  comprehensive evaluation suite designed specifically for instruction-tuned
  large language models. Unlike previous works, our evaluation involves a
  rigorous assessment of models based on problem-solving, writing ability, and
  alignment to human values. We take a holistic approach to analyze various
  factors affecting model performance, including the pretraining foundation,
  instruction-tuning data, and training methods. Our findings reveal that the
  quality of instruction data is the most crucial factor in scaling model
  performance. While open-source models demonstrate impressive writing abilities,
  there is substantial room for improvement in problem-solving and alignment. We
  are encouraged by the rapid development of models by the open-source community,
  but we also highlight the need for rigorous evaluation to support claims made
  about these models. Through INSTRUCTEVAL, we aim to foster a deeper
  understanding of instruction-tuned models and advancements in their
  capabilities. INSTRUCTEVAL is publicly available at
  https://github.com/declare-lab/instruct-eval.
paper_authors._id: 64827bfa120600dcbe532f44
paper_authors.name: Yew Ken Chia
paper_authors._id: 64827bfa120600dcbe532f45
paper_authors.name: Pengfei Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:22:39.632Z
paper_authors._id: 62c3f44617c780a4f2ed603c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657009213654-noauth.jpeg
paper_authors.fullname: Hong Pengfei
paper_authors.type: user
paper_authors.user: emrys-hong
paper_authors._id: 64827bfa120600dcbe532f46
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:23:01.390Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_authors._id: 64827bfa120600dcbe532f47
paper_authors.name: Soujanya Poria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:32:05.893Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
    Language Models
paper_summary: Instruction-tuned large language models have revolutionized natural language
  processing and have shown great potential in applications such as
  conversational agents. These models, such as GPT-4, can not only master
  language but also solve complex tasks in areas like mathematics, coding,
  medicine, and law. Despite their impressive capabilities, there is still a lack
  of comprehensive understanding regarding their full potential, primarily due to
  the black-box nature of many models and the absence of holistic evaluation
  studies. To address these challenges, we present INSTRUCTEVAL, a more
  comprehensive evaluation suite designed specifically for instruction-tuned
  large language models. Unlike previous works, our evaluation involves a
  rigorous assessment of models based on problem-solving, writing ability, and
  alignment to human values. We take a holistic approach to analyze various
  factors affecting model performance, including the pretraining foundation,
  instruction-tuning data, and training methods. Our findings reveal that the
  quality of instruction data is the most crucial factor in scaling model
  performance. While open-source models demonstrate impressive writing abilities,
  there is substantial room for improvement in problem-solving and alignment. We
  are encouraged by the rapid development of models by the open-source community,
  but we also highlight the need for rigorous evaluation to support claims made
  about these models. Through INSTRUCTEVAL, we aim to foster a deeper
  understanding of instruction-tuned models and advancements in their
  capabilities. INSTRUCTEVAL is publicly available at
  https://github.com/declare-lab/instruct-eval.
paper_ai_summary: INSTRUCTEVAL provides a comprehensive evaluation suite for instruction-tuned large language models, focusing on problem-solving, writing ability, and alignment to human values.