PlainQAFact: Automatic Factuality Evaluation Metric for Biomedical Plain
  Language Summaries Generation
Hallucinated outputs from language models pose risks in the medical domain,
especially for lay audiences making health-related decisions. Existing
factuality evaluation methods, such as entailment- and question-answering-based
(QA), struggle with plain language summary (PLS) generation due to elaborative
explanation phenomenon, which introduces external content (e.g., definitions,
background, examples) absent from the source document to enhance comprehension.
To address this, we introduce PlainQAFact, a framework trained on a
fine-grained, human-annotated dataset PlainFact, to evaluate the factuality of
both source-simplified and elaboratively explained sentences. PlainQAFact first
classifies factuality type and then assesses factuality using a
retrieval-augmented QA-based scoring method. Our approach is lightweight and
computationally efficient. Empirical results show that existing factuality
metrics fail to effectively evaluate factuality in PLS, especially for
elaborative explanations, whereas PlainQAFact achieves state-of-the-art
performance. We further analyze its effectiveness across external knowledge
sources, answer extraction strategies, overlap measures, and document
granularity levels, refining its overall factuality assessment.
67d22a337da37e254192ceac
Zhiwen You
claimed_verified
2025-03-13T08:24:27.583Z
632d21bb930d186432dfddea
/avatars/efafd9c17f2f2b69407707c560ea1237.svg
Zhiwen You
user
uzw
67d22a337da37e254192cead
Yue Guo
PlainQAFact: Automatic Factuality Evaluation Metric for Biomedical Plain
  Language Summaries Generation
Hallucinated outputs from language models pose risks in the medical domain,
especially for lay audiences making health-related decisions. Existing
factuality evaluation methods, such as entailment- and question-answering-based
(QA), struggle with plain language summary (PLS) generation due to elaborative
explanation phenomenon, which introduces external content (e.g., definitions,
background, examples) absent from the source document to enhance comprehension.
To address this, we introduce PlainQAFact, a framework trained on a
fine-grained, human-annotated dataset PlainFact, to evaluate the factuality of
both source-simplified and elaboratively explained sentences. PlainQAFact first
classifies factuality type and then assesses factuality using a
retrieval-augmented QA-based scoring method. Our approach is lightweight and
computationally efficient. Empirical results show that existing factuality
metrics fail to effectively evaluate factuality in PLS, especially for
elaborative explanations, whereas PlainQAFact achieves state-of-the-art
performance. We further analyze its effectiveness across external knowledge
sources, answer extraction strategies, overlap measures, and document
granularity levels, refining its overall factuality assessment.
PlainQAFact framework evaluates factuality in plain language summaries, outperforming existing methods by handling elaborative explanations.
entailment-based question-answering-based QA plain language summary PLS elaborative explanation elaborative explanations PlainQAFact PlainFact retrieval-augmented QA-based scoring method factuality type factuality assessment external knowledge sources answer extraction strategies overlap measures document granularity levels