title: Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for
    Large Language Models
summary: Assessing the effectiveness of large language models (LLMs) in addressing
  diverse tasks is essential for comprehending their strengths and weaknesses.
  Conventional evaluation techniques typically apply a single prompting strategy
  uniformly across datasets, not considering the varying degrees of task
  complexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomy
  that employs a Hierarchical Prompt Framework (HPF) composed of five unique
  prompting strategies, arranged from the simplest to the most complex, to assess
  LLMs more precisely and to offer a clearer perspective. This taxonomy assigns a
  score, called the Hierarchical Prompting Score (HP-Score), to datasets as well
  as LLMs based on the rules of the taxonomy, providing a nuanced understanding
  of their ability to solve diverse tasks and offering a universal measure of
  task complexity. Additionally, we introduce the Adaptive Hierarchical Prompt
  framework, which automates the selection of appropriate prompting strategies
  for each task. This study compares manual and adaptive hierarchical prompt
  frameworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,
  Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),
  IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectiveness
  of HPT, providing a reliable way to compare different tasks and LLM
  capabilities. This paper leads to the development of a universal evaluation
  metric that can be used to evaluate both the complexity of the datasets and the
  capabilities of LLMs. The implementation of both manual HPF and adaptive HPF is
  publicly available.
paper_authors._id: 66725e5e6f64b0d7be8785e1
paper_authors.name: Devichand Budagam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:45:45.142Z
paper_authors._id: 653f2d18eefdcdc9fdb446b3
paper_authors.avatarUrl: /avatars/014a1f62aabafa3dfd1d12f49d12847b.svg
paper_authors.fullname: Devichand Budagam
paper_authors.type: user
paper_authors.user: devichand
paper_authors._id: 66725e5e6f64b0d7be8785e2
paper_authors.name: Sankalp KJ
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:45:51.633Z
paper_authors._id: 664e218ea10258b399d5c358
paper_authors.avatarUrl: /avatars/77f2ee0949560504e1643263fd7084da.svg
paper_authors.fullname: Sankalp KJ
paper_authors.type: user
paper_authors.user: SankalpKJ
paper_authors._id: 66725e5e6f64b0d7be8785e3
paper_authors.name: Ashutosh Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T21:06:24.194Z
paper_authors._id: 65c7fa3a242e3ee0c656927d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c7fa3a242e3ee0c656927d/yulc-eFl2hntzfbRJWiox.jpeg
paper_authors.fullname: Ashutosh Kumar
paper_authors.type: user
paper_authors.user: ashu-1069
paper_authors._id: 66725e5e6f64b0d7be8785e4
paper_authors.name: Vinija Jain
paper_authors._id: 66725e5e6f64b0d7be8785e5
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:37:36.231Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_title: Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for
    Large Language Models
paper_summary: Assessing the effectiveness of large language models (LLMs) in addressing
  diverse tasks is essential for comprehending their strengths and weaknesses.
  Conventional evaluation techniques typically apply a single prompting strategy
  uniformly across datasets, not considering the varying degrees of task
  complexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomy
  that employs a Hierarchical Prompt Framework (HPF) composed of five unique
  prompting strategies, arranged from the simplest to the most complex, to assess
  LLMs more precisely and to offer a clearer perspective. This taxonomy assigns a
  score, called the Hierarchical Prompting Score (HP-Score), to datasets as well
  as LLMs based on the rules of the taxonomy, providing a nuanced understanding
  of their ability to solve diverse tasks and offering a universal measure of
  task complexity. Additionally, we introduce the Adaptive Hierarchical Prompt
  framework, which automates the selection of appropriate prompting strategies
  for each task. This study compares manual and adaptive hierarchical prompt
  frameworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,
  Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),
  IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectiveness
  of HPT, providing a reliable way to compare different tasks and LLM
  capabilities. This paper leads to the development of a universal evaluation
  metric that can be used to evaluate both the complexity of the datasets and the
  capabilities of LLMs. The implementation of both manual HPF and adaptive HPF is
  publicly available.
paper_ai_keywords: Hierarchical Prompting Taxonomy (HPT) Hierarchical Prompt Framework (HPF) Hierarchical Prompting Score (HP-Score) Adaptive Hierarchical Prompt framework