Exploring Precision and Recall to assess the quality and diversity of LLMs
<|reference_start|>Exploring Precision and Recall to assess the quality and diversity of LLMs: We introduce a novel evaluation framework for Large Language Models (LLMs) such as \textsc{Llama-2} and \textsc{Mistral}, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data.<|reference_end|>
@article{bronnec2024exploring,
    title={Exploring Precision and Recall to assess the quality and diversity of
  LLMs},
    author={Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann
  Chevaleyre, Alexandre Allauzen},
    journal={arXiv preprint arXiv:2402.10693},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.10693},
    primaryClass={cs.CL cs.LG}
}