title: Toward Reliable Biomedical Hypothesis Generation: Evaluating
    Truthfulness and Hallucination in Large Language Models
summary: Large language models (LLMs) have shown significant potential in scientific
  disciplines such as biomedicine, particularly in hypothesis generation, where
  they can analyze vast literature, identify patterns, and suggest research
  directions. However, a key challenge lies in evaluating the truthfulness of
  generated hypotheses, as verifying their accuracy often requires substantial
  time and resources. Additionally, the hallucination problem in LLMs can lead to
  the generation of hypotheses that appear plausible but are ultimately
  incorrect, undermining their reliability. To facilitate the systematic study of
  these challenges, we introduce TruthHypo, a benchmark for assessing the
  capabilities of LLMs in generating truthful biomedical hypotheses, and KnowHD,
  a knowledge-based hallucination detector to evaluate how well hypotheses are
  grounded in existing knowledge. Our results show that LLMs struggle to generate
  truthful hypotheses. By analyzing hallucinations in reasoning steps, we
  demonstrate that the groundedness scores provided by KnowHD serve as an
  effective metric for filtering truthful hypotheses from the diverse outputs of
  LLMs. Human evaluations further validate the utility of KnowHD in identifying
  truthful hypotheses and accelerating scientific discovery. Our data and source
  code are available at https://github.com/Teddy-XiongGZ/TruthHypo.
paper_authors._id: 683a4c9070c36271751e3fbc
paper_authors.name: Guangzhi Xiong
paper_authors._id: 683a4c9070c36271751e3fbd
paper_authors.name: Eric Xie
paper_authors._id: 683a4c9070c36271751e3fbe
paper_authors.name: Corey Williams
paper_authors._id: 683a4c9070c36271751e3fbf
paper_authors.name: Myles Kim
paper_authors._id: 683a4c9070c36271751e3fc0
paper_authors.name: Amir Hassan Shariatmadari
paper_authors._id: 683a4c9070c36271751e3fc1
paper_authors.name: Sikun Guo
paper_authors._id: 683a4c9070c36271751e3fc2
paper_authors.name: Stefan Bekiranov
paper_authors._id: 683a4c9070c36271751e3fc3
paper_authors.name: Aidong Zhang
paper_title: Toward Reliable Biomedical Hypothesis Generation: Evaluating
    Truthfulness and Hallucination in Large Language Models
paper_summary: Large language models (LLMs) have shown significant potential in scientific
  disciplines such as biomedicine, particularly in hypothesis generation, where
  they can analyze vast literature, identify patterns, and suggest research
  directions. However, a key challenge lies in evaluating the truthfulness of
  generated hypotheses, as verifying their accuracy often requires substantial
  time and resources. Additionally, the hallucination problem in LLMs can lead to
  the generation of hypotheses that appear plausible but are ultimately
  incorrect, undermining their reliability. To facilitate the systematic study of
  these challenges, we introduce TruthHypo, a benchmark for assessing the
  capabilities of LLMs in generating truthful biomedical hypotheses, and KnowHD,
  a knowledge-based hallucination detector to evaluate how well hypotheses are
  grounded in existing knowledge. Our results show that LLMs struggle to generate
  truthful hypotheses. By analyzing hallucinations in reasoning steps, we
  demonstrate that the groundedness scores provided by KnowHD serve as an
  effective metric for filtering truthful hypotheses from the diverse outputs of
  LLMs. Human evaluations further validate the utility of KnowHD in identifying
  truthful hypotheses and accelerating scientific discovery. Our data and source
  code are available at https://github.com/Teddy-XiongGZ/TruthHypo.
paper_ai_summary: TruthHypo and KnowHD assess and filter truthful biomedical hypotheses generated by large language models, addressing the challenge of hallucination and truthfulness in scientific research.
paper_ai_keywords: large language models LLMs hypothesis generation truthfulness hallucination biomedical hypotheses benchmark knowledge-based hallucination detector groundedness scores human evaluations