Evaluating Large Language Models with fmeval
<|reference_start|>Evaluating Large Language Models with fmeval: fmeval is an open source library to evaluate large language models (LLMs) in a range of tasks. It helps practitioners evaluate their model for task performance and along multiple responsible AI dimensions. This paper presents the library and exposes its underlying design principles: simplicity, coverage, extensibility and performance. We then present how these were implemented in the scientific and engineering choices taken when developing fmeval. A case study demonstrates a typical use case for the library: picking a suitable model for a question answering task. We close by discussing limitations and further work in the development of the library. fmeval can be found at https://github.com/aws/fmeval.<|reference_end|>
@article{schwöbel2024evaluating,
    title={Evaluating Large Language Models with fmeval},
    author={Pola Schw"obel, Luca Franceschi, Muhammad Bilal Zafar, Keerthan
  Vasist, Aman Malhotra, Tomer Shenhar, Pinal Tailor, Pinar Yilmaz, Michael
  Diamond, Michele Donini},
    journal={arXiv preprint arXiv:2407.12872},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12872},
    primaryClass={cs.CL cs.LG}
}