title: CDM: A Reliable Metric for Fair and Accurate Formula Recognition
    Evaluation
summary: Formula recognition presents significant challenges due to the complicated
  structure and varied notation of mathematical expressions. Despite continuous
  advancements in formula recognition models, the evaluation metrics employed by
  these models, such as BLEU and Edit Distance, still exhibit notable
  limitations. They overlook the fact that the same formula has diverse
  representations and is highly sensitive to the distribution of training data,
  thereby causing the unfairness in formula recognition evaluation. To this end,
  we propose a Character Detection Matching (CDM) metric, ensuring the evaluation
  objectivity by designing a image-level rather than LaTex-level metric score.
  Specifically, CDM renders both the model-predicted LaTeX and the ground-truth
  LaTeX formulas into image-formatted formulas, then employs visual feature
  extraction and localization techniques for precise character-level matching,
  incorporating spatial position information. Such a spatially-aware and
  character-matching method offers a more accurate and equitable evaluation
  compared with previous BLEU and Edit Distance metrics that rely solely on
  text-based character matching. Experimentally, we evaluated various formula
  recognition models using CDM, BLEU, and ExpRate metrics. Their results
  demonstrate that the CDM aligns more closely with human evaluation standards
  and provides a fairer comparison across different models by eliminating
  discrepancies caused by diverse formula representations.
paper_authors._id: 66dab2f7006a9245b9284a70
paper_authors.name: Bin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T09:06:04.441Z
paper_authors._id: 63ae9ff5557befe297a76f90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: wanderkid
paper_authors._id: 66dab2f7006a9245b9284a71
paper_authors.name: Fan Wu
paper_authors._id: 66dab2f7006a9245b9284a72
paper_authors.name: Linke Ouyang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:24:45.266Z
paper_authors._id: 66753c556f2ac48ee625d7d1
paper_authors.avatarUrl: /avatars/8f7c252675fd8a096794d12971903722.svg
paper_authors.fullname: Linke Ouyang
paper_authors.type: user
paper_authors.user: ouyanglinke
paper_authors._id: 66dab2f7006a9245b9284a73
paper_authors.name: Zhuangcheng Gu
paper_authors._id: 66dab2f7006a9245b9284a74
paper_authors.name: Rui Zhang
paper_authors._id: 66dab2f7006a9245b9284a75
paper_authors.name: Renqiu Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:23:41.774Z
paper_authors._id: 65b7ae76768464877cdb2e39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b7ae76768464877cdb2e39/uAWPo4tpkqbZoDeEkc7y0.jpeg
paper_authors.fullname: Renqiu Xia
paper_authors.type: user
paper_authors.user: renqiux0302
paper_authors._id: 66dab2f7006a9245b9284a76
paper_authors.name: Bo Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:23:35.959Z
paper_authors._id: 643dfd235aafbdca3a5792c0
paper_authors.avatarUrl: /avatars/ce8553cf5936012c692e08054ee27937.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: BoZhang
paper_authors._id: 66dab2f7006a9245b9284a77
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:22:58.520Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_title: CDM: A Reliable Metric for Fair and Accurate Formula Recognition
    Evaluation
paper_summary: Formula recognition presents significant challenges due to the complicated
  structure and varied notation of mathematical expressions. Despite continuous
  advancements in formula recognition models, the evaluation metrics employed by
  these models, such as BLEU and Edit Distance, still exhibit notable
  limitations. They overlook the fact that the same formula has diverse
  representations and is highly sensitive to the distribution of training data,
  thereby causing the unfairness in formula recognition evaluation. To this end,
  we propose a Character Detection Matching (CDM) metric, ensuring the evaluation
  objectivity by designing a image-level rather than LaTex-level metric score.
  Specifically, CDM renders both the model-predicted LaTeX and the ground-truth
  LaTeX formulas into image-formatted formulas, then employs visual feature
  extraction and localization techniques for precise character-level matching,
  incorporating spatial position information. Such a spatially-aware and
  character-matching method offers a more accurate and equitable evaluation
  compared with previous BLEU and Edit Distance metrics that rely solely on
  text-based character matching. Experimentally, we evaluated various formula
  recognition models using CDM, BLEU, and ExpRate metrics. Their results
  demonstrate that the CDM aligns more closely with human evaluation standards
  and provides a fairer comparison across different models by eliminating
  discrepancies caused by diverse formula representations.
paper_ai_keywords: Character Detection Matching (CDM) BLEU Edit Distance LaTeX visual feature extraction localization techniques character-level matching spatial position information ExpRate