title: FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
    Sets
summary: Evaluation of Large Language Models (LLMs) is challenging because aligning to
  human values requires the composition of multiple skills and the required set
  of skills varies depending on the instruction. Recent studies have evaluated
  the performance of LLMs in two ways, (1) automatic evaluation on several
  independent benchmarks and (2) human or machined-based evaluation giving an
  overall score to the response. However, both settings are coarse-grained
  evaluations, not considering the nature of user instructions that require
  instance-wise skill composition, which limits the interpretation of the true
  capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
  Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
  protocol that can be used for both model-based and human-based evaluation which
  decomposes coarse-level scoring to an instance-wise skill set-level.
  Specifically, we define 12 fine-grained skills needed for LLMs to follow
  open-ended user instructions and construct an evaluation set by allocating a
  set of skills for each instance. Additionally, by annotating the target domains
  and difficulty level for each instance, FLASK provides a holistic view with a
  comprehensive analysis of a model's performance depending on skill, domain, and
  difficulty. Through using FLASK, we compare multiple open-sourced and
  proprietary LLMs and observe highly-correlated findings between model-based and
  human-based evaluations. FLASK enables developers to more accurately measure
  the model performance and how it can be improved by analyzing factors that make
  LLMs proficient in particular skills. For practitioners, FLASK can be used to
  recommend suitable models for particular situations through comprehensive
  comparison among various LLMs. We release the evaluation data and code
  implementation at https://github.com/kaistAI/FLASK.
paper_authors._id: 64ba0fc1b3bd26c44b114b8d
paper_authors.name: Seonghyeon Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T07:59:37.015Z
paper_authors._id: 62551f7767f0b85962624047
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
paper_authors.fullname: Seonghyeon Ye
paper_authors.type: user
paper_authors.user: seonghyeonye
paper_authors._id: 64ba0fc1b3bd26c44b114b8e
paper_authors.name: Doyoung Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T07:59:59.393Z
paper_authors._id: 610659fc377b3983fe49b3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665487693252-610659fc377b3983fe49b3a7.jpeg
paper_authors.fullname: doyoung kim
paper_authors.type: user
paper_authors.user: doyoungkim
paper_authors._id: 64ba0fc1b3bd26c44b114b8f
paper_authors.name: Sungdong Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:00:18.381Z
paper_authors._id: 6084ebe956f7623a0b849153
paper_authors.avatarUrl: /avatars/c56fb056800c145e5a8234e27b6b5a31.svg
paper_authors.fullname: Sungdong kim
paper_authors.type: user
paper_authors.user: dsksd
paper_authors._id: 64ba0fc1b3bd26c44b114b90
paper_authors.name: Hyeonbin Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:00:32.770Z
paper_authors._id: 647eaaf61a1fcad2fdc5d1ef
paper_authors.avatarUrl: /avatars/596d7a61d6e6227d38b661210a32fed0.svg
paper_authors.fullname: Hyeonbin Hwang 
paper_authors.type: user
paper_authors.user: hbin0701
paper_authors._id: 64ba0fc1b3bd26c44b114b91
paper_authors.name: Seungone Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:01:11.845Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 64ba0fc1b3bd26c44b114b92
paper_authors.name: Yongrae Jo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:01:26.466Z
paper_authors._id: 649e06313e5e7504763dfe03
paper_authors.avatarUrl: /avatars/1c4d19de5f2950d3342480c4b3e01047.svg
paper_authors.fullname: Yongrae Jo
paper_authors.type: user
paper_authors.user: dreamgonfly
paper_authors._id: 64ba0fc1b3bd26c44b114b93
paper_authors.name: James Thorne
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:01:59.704Z
paper_authors._id: 63b54db5889aa6707f08f2d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b54db5889aa6707f08f2d1/zUnyx-8Zabxr54poZtUo4.jpeg
paper_authors.fullname: James Thorne
paper_authors.type: user
paper_authors.user: j6mes
paper_authors._id: 64ba0fc1b3bd26c44b114b94
paper_authors.name: Juho Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:02:24.631Z
paper_authors._id: 62d6411d4d766e27a4736782
paper_authors.avatarUrl: /avatars/16f259d27134beafdd28620046bdfcc6.svg
paper_authors.fullname: Juho Kim
paper_authors.type: user
paper_authors.user: mcpanic
paper_authors._id: 64ba0fc1b3bd26c44b114b95
paper_authors.name: Minjoon Seo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:02:43.071Z
paper_authors._id: 621f05ba970615ad5861ceb1
paper_authors.avatarUrl: /avatars/7e1902aa71369a524afda9b0a9e88e22.svg
paper_authors.fullname: Minjoon Seo
paper_authors.type: user
paper_authors.user: minjoon
paper_title: FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
    Sets
paper_summary: Evaluation of Large Language Models (LLMs) is challenging because aligning to
  human values requires the composition of multiple skills and the required set
  of skills varies depending on the instruction. Recent studies have evaluated
  the performance of LLMs in two ways, (1) automatic evaluation on several
  independent benchmarks and (2) human or machined-based evaluation giving an
  overall score to the response. However, both settings are coarse-grained
  evaluations, not considering the nature of user instructions that require
  instance-wise skill composition, which limits the interpretation of the true
  capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
  Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
  protocol that can be used for both model-based and human-based evaluation which
  decomposes coarse-level scoring to an instance-wise skill set-level.
  Specifically, we define 12 fine-grained skills needed for LLMs to follow
  open-ended user instructions and construct an evaluation set by allocating a
  set of skills for each instance. Additionally, by annotating the target domains
  and difficulty level for each instance, FLASK provides a holistic view with a
  comprehensive analysis of a model's performance depending on skill, domain, and
  difficulty. Through using FLASK, we compare multiple open-sourced and
  proprietary LLMs and observe highly-correlated findings between model-based and
  human-based evaluations. FLASK enables developers to more accurately measure
  the model performance and how it can be improved by analyzing factors that make
  LLMs proficient in particular skills. For practitioners, FLASK can be used to
  recommend suitable models for particular situations through comprehensive
  comparison among various LLMs. We release the evaluation data and code
  implementation at https://github.com/kaistAI/FLASK.
paper_ai_keywords: fine-grained evaluation alignment skill sets open-ended user instructions skill set decomposition target domains difficulty level model-based evaluation human-based evaluation comprehensive analysis model performance measurement model improvement analysis skill proficiency model recommendation