Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for
  Large Language Models
Assessing the effectiveness of large language models (LLMs) in addressing
diverse tasks is essential for comprehending their strengths and weaknesses.
Conventional evaluation techniques typically apply a single prompting strategy
uniformly across datasets, not considering the varying degrees of task
complexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomy
that employs a Hierarchical Prompt Framework (HPF) composed of five unique
prompting strategies, arranged from the simplest to the most complex, to assess
LLMs more precisely and to offer a clearer perspective. This taxonomy assigns a
score, called the Hierarchical Prompting Score (HP-Score), to datasets as well
as LLMs based on the rules of the taxonomy, providing a nuanced understanding
of their ability to solve diverse tasks and offering a universal measure of
task complexity. Additionally, we introduce the Adaptive Hierarchical Prompt
framework, which automates the selection of appropriate prompting strategies
for each task. This study compares manual and adaptive hierarchical prompt
frameworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,
Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),
IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectiveness
of HPT, providing a reliable way to compare different tasks and LLM
capabilities. This paper leads to the development of a universal evaluation
metric that can be used to evaluate both the complexity of the datasets and the
capabilities of LLMs. The implementation of both manual HPF and adaptive HPF is
publicly available.
66725e5e6f64b0d7be8785e1
Devichand Budagam
admin_assigned
2024-06-20T07:45:45.142Z
653f2d18eefdcdc9fdb446b3
/avatars/014a1f62aabafa3dfd1d12f49d12847b.svg
Devichand Budagam
user
devichand
66725e5e6f64b0d7be8785e2
Sankalp KJ
admin_assigned
2024-06-20T07:45:51.633Z
664e218ea10258b399d5c358
/avatars/77f2ee0949560504e1643263fd7084da.svg
Sankalp KJ
user
SankalpKJ
66725e5e6f64b0d7be8785e3
Ashutosh Kumar
claimed_verified
2025-01-29T21:06:24.194Z
65c7fa3a242e3ee0c656927d
https://cdn-avatars.huggingface.co/v1/production/uploads/65c7fa3a242e3ee0c656927d/yulc-eFl2hntzfbRJWiox.jpeg
Ashutosh Kumar
user
ashu-1069
66725e5e6f64b0d7be8785e4
Vinija Jain
66725e5e6f64b0d7be8785e5
Aman Chadha
claimed_verified
2024-06-19T07:37:36.231Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for
  Large Language Models
Assessing the effectiveness of large language models (LLMs) in addressing
diverse tasks is essential for comprehending their strengths and weaknesses.
Conventional evaluation techniques typically apply a single prompting strategy
uniformly across datasets, not considering the varying degrees of task
complexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomy
that employs a Hierarchical Prompt Framework (HPF) composed of five unique
prompting strategies, arranged from the simplest to the most complex, to assess
LLMs more precisely and to offer a clearer perspective. This taxonomy assigns a
score, called the Hierarchical Prompting Score (HP-Score), to datasets as well
as LLMs based on the rules of the taxonomy, providing a nuanced understanding
of their ability to solve diverse tasks and offering a universal measure of
task complexity. Additionally, we introduce the Adaptive Hierarchical Prompt
framework, which automates the selection of appropriate prompting strategies
for each task. This study compares manual and adaptive hierarchical prompt
frameworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,
Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),
IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectiveness
of HPT, providing a reliable way to compare different tasks and LLM
capabilities. This paper leads to the development of a universal evaluation
metric that can be used to evaluate both the complexity of the datasets and the
capabilities of LLMs. The implementation of both manual HPF and adaptive HPF is
publicly available.
The Hierarchical Prompting Taxonomy (HPT) introduces a system of five prompting strategies to evaluate large language models (LLMs) more precisely across different tasks, providing a universal measure of task complexity and model capability.
Hierarchical Prompting Taxonomy Hierarchical Prompt Framework HP-Score Adaptive Hierarchical Prompt framework instruction-tuned LLMs boolQ CommonSenseQA IWSLT-2017 en-fr SamSum