title: MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations
    in Large Language Models
summary: Advancements in Large Language Models (LLMs) and their increasing use in
  medical question-answering necessitate rigorous evaluation of their
  reliability. A critical challenge lies in hallucination, where models generate
  plausible yet factually incorrect outputs. In the medical domain, this poses
  serious risks to patient safety and clinical decision-making. To address this,
  we introduce MedHallu, the first benchmark specifically designed for medical
  hallucination detection. MedHallu comprises 10,000 high-quality question-answer
  pairs derived from PubMedQA, with hallucinated answers systematically generated
  through a controlled pipeline. Our experiments show that state-of-the-art LLMs,
  including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical,
  struggle with this binary hallucination detection task, with the best model
  achieving an F1 score as low as 0.625 for detecting "hard" category
  hallucinations. Using bidirectional entailment clustering, we show that
  harder-to-detect hallucinations are semantically closer to ground truth.
  Through experiments, we also show incorporating domain-specific knowledge and
  introducing a "not sure" category as one of the answer categories improves the
  precision and F1 scores by up to 38% relative to baselines.
paper_authors._id: 67b7e1a0e5dcadcedba108e8
paper_authors.name: Shrey Pandit
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-22T19:42:39.549Z
paper_authors._id: 648749094dea003c6dae810f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648749094dea003c6dae810f/gHUHSBt1zrt8wjO1YwTNu.jpeg
paper_authors.fullname: Shrey Pandit
paper_authors.type: user
paper_authors.user: SP2001
paper_authors._id: 67b7e1a0e5dcadcedba108e9
paper_authors.name: Jiawei Xu
paper_authors._id: 67b7e1a0e5dcadcedba108ea
paper_authors.name: Junyuan Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:58:00.251Z
paper_authors._id: 6400cf982b67d27affce2d89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6400cf982b67d27affce2d89/Vs042d2M-iV2wk9Q_Jqh3.jpeg
paper_authors.fullname: Junyuan Hong
paper_authors.type: user
paper_authors.user: jyhong836
paper_authors._id: 67b7e1a0e5dcadcedba108eb
paper_authors.name: Zhangyang Wang
paper_authors._id: 67b7e1a0e5dcadcedba108ec
paper_authors.name: Tianlong Chen
paper_authors._id: 67b7e1a0e5dcadcedba108ed
paper_authors.name: Kaidi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:00:54.855Z
paper_authors._id: 65aaa5b4d2adc31ee3eab350
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65aaa5b4d2adc31ee3eab350/vzDrR7JVCEShn_Vk-NCgT.jpeg
paper_authors.fullname: Kaidi Xu
paper_authors.type: user
paper_authors.user: KaidiXu1
paper_authors._id: 67b7e1a0e5dcadcedba108ee
paper_authors.name: Ying Ding
paper_title: MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations
    in Large Language Models
paper_summary: Advancements in Large Language Models (LLMs) and their increasing use in
  medical question-answering necessitate rigorous evaluation of their
  reliability. A critical challenge lies in hallucination, where models generate
  plausible yet factually incorrect outputs. In the medical domain, this poses
  serious risks to patient safety and clinical decision-making. To address this,
  we introduce MedHallu, the first benchmark specifically designed for medical
  hallucination detection. MedHallu comprises 10,000 high-quality question-answer
  pairs derived from PubMedQA, with hallucinated answers systematically generated
  through a controlled pipeline. Our experiments show that state-of-the-art LLMs,
  including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical,
  struggle with this binary hallucination detection task, with the best model
  achieving an F1 score as low as 0.625 for detecting "hard" category
  hallucinations. Using bidirectional entailment clustering, we show that
  harder-to-detect hallucinations are semantically closer to ground truth.
  Through experiments, we also show incorporating domain-specific knowledge and
  introducing a "not sure" category as one of the answer categories improves the
  precision and F1 scores by up to 38% relative to baselines.
paper_ai_keywords: Large Language Models (LLMs) medical question-answering hallucination MedHallu PubMedQA bidirectional entailment clustering F1 score medically fine-tuned UltraMedical "not sure" category