title: RaTEScore: A Metric for Radiology Report Generation
summary: This paper introduces a novel, entity-aware metric, termed as Radiological
  Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports
  generated by AI models. RaTEScore emphasizes crucial medical entities such as
  diagnostic outcomes and anatomical details, and is robust against complex
  medical synonyms and sensitive to negation expressions. Technically, we
  developed a comprehensive medical NER dataset, RaTE-NER, and trained an NER
  model specifically for this purpose. This model enables the decomposition of
  complex radiological reports into constituent medical entities. The metric
  itself is derived by comparing the similarity of entity embeddings, obtained
  from a language model, based on their types and relevance to clinical
  significance. Our evaluations demonstrate that RaTEScore aligns more closely
  with human preference than existing metrics, validated both on established
  public benchmarks and our newly proposed RaTE-Eval benchmark.
paper_authors._id: 667a32e17dc34fe17cef3a84
paper_authors.name: Weike Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-01T07:47:05.570Z
paper_authors._id: 64365addfae287005149dd24
paper_authors.avatarUrl: /avatars/bd6d4512d66fd9fd7fd5476ea7a44b46.svg
paper_authors.fullname: Weike Zhao
paper_authors.type: user
paper_authors.user: Angelakeke
paper_authors._id: 667a32e17dc34fe17cef3a85
paper_authors.name: Chaoyi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T15:04:29.973Z
paper_authors._id: 6436aaaa0c77d7c5036abdbd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6436aaaa0c77d7c5036abdbd/C9A276yEeAPkKLEqUcjl_.jpeg
paper_authors.fullname: Chaoyi Wu
paper_authors.type: user
paper_authors.user: chaoyi-wu
paper_authors._id: 667a32e17dc34fe17cef3a86
paper_authors.name: Xiaoman Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T15:04:37.223Z
paper_authors._id: 664dbb92724eaeb47777ded9
paper_authors.avatarUrl: /avatars/f507ebddee54b35c14c96658c1858428.svg
paper_authors.fullname: Xiaoman Zhang
paper_authors.type: user
paper_authors.user: XiaomanZhang
paper_authors._id: 667a32e17dc34fe17cef3a87
paper_authors.name: Ya Zhang
paper_authors._id: 667a32e17dc34fe17cef3a88
paper_authors.name: Yanfeng Wang
paper_authors._id: 667a32e17dc34fe17cef3a89
paper_authors.name: Weidi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T15:05:32.447Z
paper_authors._id: 63f1841f2f7c0152e8748ec4
paper_authors.avatarUrl: /avatars/5398658c7d0cf556531a625a4ca5d18a.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Weidi
paper_title: RaTEScore: A Metric for Radiology Report Generation
paper_summary: This paper introduces a novel, entity-aware metric, termed as Radiological
  Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports
  generated by AI models. RaTEScore emphasizes crucial medical entities such as
  diagnostic outcomes and anatomical details, and is robust against complex
  medical synonyms and sensitive to negation expressions. Technically, we
  developed a comprehensive medical NER dataset, RaTE-NER, and trained an NER
  model specifically for this purpose. This model enables the decomposition of
  complex radiological reports into constituent medical entities. The metric
  itself is derived by comparing the similarity of entity embeddings, obtained
  from a language model, based on their types and relevance to clinical
  significance. Our evaluations demonstrate that RaTEScore aligns more closely
  with human preference than existing metrics, validated both on established
  public benchmarks and our newly proposed RaTE-Eval benchmark.
paper_ai_keywords: entity-aware metric Radiological Report (Text) Evaluation (RaTEScore) diagnostic outcomes anatomical details NER dataset RaTE-NER NER model entity embeddings language model clinical significance RaTE-Eval benchmark