Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
<|reference_start|>Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications: The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work.<|reference_end|>
@article{arabzadeh2024towards,
    title={Towards better Human-Agent Alignment: Assessing Task Utility in
  LLM-Powered Applications},
    author={Negar Arabzadeh and Julia Kiseleva and Qingyun Wu and Chi Wang and
  Ahmed Awadallah and Victor Dibia and Adam Fourney and Charles Clarke},
    journal={arXiv preprint arXiv:2402.09015},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.09015},
    primaryClass={cs.CL cs.AI}
}