RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain
<|reference_start|>RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain: Large Language Models (LLMs) increasingly support applications in a wide range of domains, some with potential high societal impact such as biomedicine, yet their reliability in realistic use cases is under-researched. In this work we introduce the Reliability AssesMent for Biomedical LLM Assistants (RAmBLA) framework and evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain. We identify prompt robustness, high recall, and a lack of hallucinations as necessary criteria for this use case. We design shortform tasks and tasks requiring LLM freeform responses mimicking real-world user interactions. We evaluate LLM performance using semantic similarity with a ground truth response, through an evaluator LLM.<|reference_end|>
@article{bolton2024rambla:,
    title={RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants
  in the Biomedical Domain},
    author={William James Bolton, Rafael Poyiadzi, Edward R. Morrell, Gabriela van
  Bergen Gonzalez Bueno, Lea Goetz},
    journal={arXiv preprint arXiv:2403.14578},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14578},
    primaryClass={cs.LG cs.AI}
}