MedConceptsQA: Open Source Medical Concepts QA Benchmark
<|reference_start|>MedConceptsQA: Open Source Medical Concepts QA Benchmark: We present MedConceptsQA, a dedicated open source benchmark for medical concepts question answering. The benchmark comprises of questions of various medical concepts across different vocabularies: diagnoses, procedures, and drugs. The questions are categorized into three levels of difficulty: easy, medium, and hard. We conducted evaluations of the benchmark using various Large Language Models. Our findings show that pre-trained clinical Large Language Models achieved accuracy levels close to random guessing on this benchmark, despite being pre-trained on medical data. However, GPT-4 achieves an absolute average improvement of nearly 27%-37% (27% for zero-shot learning and 37% for few-shot learning) when compared to clinical Large Language Models. Our benchmark serves as a valuable resource for evaluating the understanding and reasoning of medical concepts by Large Language Models. Our benchmark is available at https://huggingface.co/datasets/ofir408/MedConceptsQA<|reference_end|>
@article{shoham2024medconceptsqa:,
    title={MedConceptsQA: Open Source Medical Concepts QA Benchmark},
    author={Ofir Ben Shoham, Nadav Rappoport},
    journal={arXiv preprint arXiv:2405.07348},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07348},
    primaryClass={cs.CL cs.LG}
}