title: What are the Desired Characteristics of Calibration Sets? Identifying
    Correlates on Long Form Scientific Summarization
summary: Summarization models often generate text that is poorly calibrated to quality
  metrics because they are trained to maximize the likelihood of a single
  reference (MLE). To address this, recent work has added a calibration step,
  which exposes a model to its own ranked outputs to improve relevance or, in a
  separate line of work, contrasts positive and negative sets to improve
  faithfulness. While effective, much of this work has focused on how to generate
  and optimize these sets. Less is known about why one setup is more effective
  than another. In this work, we uncover the underlying characteristics of
  effective sets. For each training instance, we form a large, diverse pool of
  candidates and systematically vary the subsets used for calibration
  fine-tuning. Each selection strategy targets distinct aspects of the sets, such
  as lexical diversity or the size of the gap between positive and negatives. On
  three diverse scientific long-form summarization datasets (spanning biomedical,
  clinical, and chemical domains), we find, among others, that faithfulness
  calibration is optimal when the negative sets are extractive and more likely to
  be generated, whereas for relevance calibration, the metric margin between
  candidates should be maximized and surprise--the disagreement between model and
  metric defined candidate rankings--minimized. Code to create, select, and
  optimize calibration sets is available at
  https://github.com/griff4692/calibrating-summaries
paper_authors._id: 6461adbd9aad9e14972691fa
paper_authors.name: Griffin Adams
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-15T12:50:27.211Z
paper_authors._id: 611693254ef9fdfbf45dc4d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628869402886-noauth.jpeg
paper_authors.fullname: Griffin Adams
paper_authors.type: user
paper_authors.user: griffin
paper_authors._id: 6461adbd9aad9e14972691fb
paper_authors.name: Bichlien H Nguyen
paper_authors._id: 6461adbd9aad9e14972691fc
paper_authors.name: Jake Smith
paper_authors._id: 6461adbd9aad9e14972691fd
paper_authors.name: Yingce Xia
paper_authors._id: 6461adbd9aad9e14972691fe
paper_authors.name: Shufang Xie
paper_authors._id: 6461adbd9aad9e14972691ff
paper_authors.name: Anna Ostropolets
paper_authors._id: 6461adbd9aad9e1497269200
paper_authors.name: Budhaditya Deb
paper_authors._id: 6461adbd9aad9e1497269201
paper_authors.name: Yuan-Jyue Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-25T21:32:59.331Z
paper_authors._id: 6601ec932fb1cc3fecb29491
paper_authors.avatarUrl: /avatars/034006e82142f6f214891ffa23f89d9f.svg
paper_authors.fullname: Yuan-Jyue Chen
paper_authors.type: user
paper_authors.user: yuanjc
paper_authors._id: 6461adbd9aad9e1497269202
paper_authors.name: Tristan Naumann
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-02T17:36:51.417Z
paper_authors._id: 5e5870466bc35159a08ca572
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
paper_authors.fullname: Tristan Naumann
paper_authors.type: user
paper_authors.user: tnaumann
paper_authors._id: 6461adbd9aad9e1497269203
paper_authors.name: Noémie Elhadad
paper_title: What are the Desired Characteristics of Calibration Sets? Identifying
    Correlates on Long Form Scientific Summarization
paper_summary: Summarization models often generate text that is poorly calibrated to quality
  metrics because they are trained to maximize the likelihood of a single
  reference (MLE). To address this, recent work has added a calibration step,
  which exposes a model to its own ranked outputs to improve relevance or, in a
  separate line of work, contrasts positive and negative sets to improve
  faithfulness. While effective, much of this work has focused on how to generate
  and optimize these sets. Less is known about why one setup is more effective
  than another. In this work, we uncover the underlying characteristics of
  effective sets. For each training instance, we form a large, diverse pool of
  candidates and systematically vary the subsets used for calibration
  fine-tuning. Each selection strategy targets distinct aspects of the sets, such
  as lexical diversity or the size of the gap between positive and negatives. On
  three diverse scientific long-form summarization datasets (spanning biomedical,
  clinical, and chemical domains), we find, among others, that faithfulness
  calibration is optimal when the negative sets are extractive and more likely to
  be generated, whereas for relevance calibration, the metric margin between
  candidates should be maximized and surprise--the disagreement between model and
  metric defined candidate rankings--minimized. Code to create, select, and
  optimize calibration sets is available at
  https://github.com/griff4692/calibrating-summaries
paper_ai_keywords: likelihood multimodal MLE labels calibration relevance faithfulness negatives subsets training instance pool fine-tuning lexical system margin surprise disagreement candidate rankings summarization datasets biomedical clinical chemical gold descriptions