title: PlainQAFact: Automatic Factuality Evaluation Metric for Biomedical Plain
    Language Summaries Generation
summary: Hallucinated outputs from language models pose risks in the medical domain,
  especially for lay audiences making health-related decisions. Existing
  factuality evaluation methods, such as entailment- and question-answering-based
  (QA), struggle with plain language summary (PLS) generation due to elaborative
  explanation phenomenon, which introduces external content (e.g., definitions,
  background, examples) absent from the source document to enhance comprehension.
  To address this, we introduce PlainQAFact, a framework trained on a
  fine-grained, human-annotated dataset PlainFact, to evaluate the factuality of
  both source-simplified and elaboratively explained sentences. PlainQAFact first
  classifies factuality type and then assesses factuality using a
  retrieval-augmented QA-based scoring method. Our approach is lightweight and
  computationally efficient. Empirical results show that existing factuality
  metrics fail to effectively evaluate factuality in PLS, especially for
  elaborative explanations, whereas PlainQAFact achieves state-of-the-art
  performance. We further analyze its effectiveness across external knowledge
  sources, answer extraction strategies, overlap measures, and document
  granularity levels, refining its overall factuality assessment.
paper_authors._id: 67d22a337da37e254192ceac
paper_authors.name: Zhiwen You
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:24:27.583Z
paper_authors._id: 632d21bb930d186432dfddea
paper_authors.avatarUrl: /avatars/efafd9c17f2f2b69407707c560ea1237.svg
paper_authors.fullname: Zhiwen You
paper_authors.type: user
paper_authors.user: uzw
paper_authors._id: 67d22a337da37e254192cead
paper_authors.name: Yue Guo
paper_title: PlainQAFact: Automatic Factuality Evaluation Metric for Biomedical Plain
    Language Summaries Generation
paper_summary: Hallucinated outputs from language models pose risks in the medical domain,
  especially for lay audiences making health-related decisions. Existing
  factuality evaluation methods, such as entailment- and question-answering-based
  (QA), struggle with plain language summary (PLS) generation due to elaborative
  explanation phenomenon, which introduces external content (e.g., definitions,
  background, examples) absent from the source document to enhance comprehension.
  To address this, we introduce PlainQAFact, a framework trained on a
  fine-grained, human-annotated dataset PlainFact, to evaluate the factuality of
  both source-simplified and elaboratively explained sentences. PlainQAFact first
  classifies factuality type and then assesses factuality using a
  retrieval-augmented QA-based scoring method. Our approach is lightweight and
  computationally efficient. Empirical results show that existing factuality
  metrics fail to effectively evaluate factuality in PLS, especially for
  elaborative explanations, whereas PlainQAFact achieves state-of-the-art
  performance. We further analyze its effectiveness across external knowledge
  sources, answer extraction strategies, overlap measures, and document
  granularity levels, refining its overall factuality assessment.
paper_ai_summary: PlainQAFact framework evaluates factuality in plain language summaries, outperforming existing methods by handling elaborative explanations.
paper_ai_keywords: entailment-based question-answering-based QA plain language summary PLS elaborative explanation elaborative explanations PlainQAFact PlainFact retrieval-augmented QA-based scoring method factuality type factuality assessment external knowledge sources answer extraction strategies overlap measures document granularity levels