Assessing and Verifying Task Utility in LLM-Powered Applications
<|reference_start|>Assessing and Verifying Task Utility in LLM-Powered Applications: The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks. However, a significant gap remains in assessing to what extent LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the need to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the effectiveness and robustness of AgentEval for two open source datasets including Math Problem solving and ALFWorld House-hold related tasks. For reproducibility purposes, we make the data, code and all the logs publicly available at https://bit.ly/3w3yKcS .<|reference_end|>
@article{arabzadeh2024assessing,
    title={Assessing and Verifying Task Utility in LLM-Powered Applications},
    author={Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qinqyun Wu, Chi Wang, Ahmed
  Awadallah, Charles L. A. Clarke and Julia Kiseleva},
    journal={arXiv preprint arXiv:2405.02178},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.02178},
    primaryClass={cs.CL cs.AI}
}