What are the Desired Characteristics of Calibration Sets? Identifying
  Correlates on Long Form Scientific Summarization
Summarization models often generate text that is poorly calibrated to quality
metrics because they are trained to maximize the likelihood of a single
reference (MLE). To address this, recent work has added a calibration step,
which exposes a model to its own ranked outputs to improve relevance or, in a
separate line of work, contrasts positive and negative sets to improve
faithfulness. While effective, much of this work has focused on how to generate
and optimize these sets. Less is known about why one setup is more effective
than another. In this work, we uncover the underlying characteristics of
effective sets. For each training instance, we form a large, diverse pool of
candidates and systematically vary the subsets used for calibration
fine-tuning. Each selection strategy targets distinct aspects of the sets, such
as lexical diversity or the size of the gap between positive and negatives. On
three diverse scientific long-form summarization datasets (spanning biomedical,
clinical, and chemical domains), we find, among others, that faithfulness
calibration is optimal when the negative sets are extractive and more likely to
be generated, whereas for relevance calibration, the metric margin between
candidates should be maximized and surprise--the disagreement between model and
metric defined candidate rankings--minimized. Code to create, select, and
optimize calibration sets is available at
https://github.com/griff4692/calibrating-summaries
6461adbd9aad9e14972691fa
Griffin Adams
extracted_confirmed
2023-05-15T12:50:27.211Z
611693254ef9fdfbf45dc4d6
https://cdn-avatars.huggingface.co/v1/production/uploads/1628869402886-noauth.jpeg
Griffin Adams
user
griffin
6461adbd9aad9e14972691fb
Bichlien H Nguyen
6461adbd9aad9e14972691fc
Jake Smith
6461adbd9aad9e14972691fd
Yingce Xia
6461adbd9aad9e14972691fe
Shufang Xie
6461adbd9aad9e14972691ff
Anna Ostropolets
6461adbd9aad9e1497269200
Budhaditya Deb
6461adbd9aad9e1497269201
Yuan-Jyue Chen
extracted_pending
2024-03-25T21:32:59.331Z
6601ec932fb1cc3fecb29491
/avatars/034006e82142f6f214891ffa23f89d9f.svg
Yuan-Jyue Chen
user
yuanjc
6461adbd9aad9e1497269202
Tristan Naumann
extracted_confirmed
2023-06-02T17:36:51.417Z
5e5870466bc35159a08ca572
https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
Tristan Naumann
user
tnaumann
6461adbd9aad9e1497269203
Noémie Elhadad
What are the Desired Characteristics of Calibration Sets? Identifying
  Correlates on Long Form Scientific Summarization
Summarization models often generate text that is poorly calibrated to quality
metrics because they are trained to maximize the likelihood of a single
reference (MLE). To address this, recent work has added a calibration step,
which exposes a model to its own ranked outputs to improve relevance or, in a
separate line of work, contrasts positive and negative sets to improve
faithfulness. While effective, much of this work has focused on how to generate
and optimize these sets. Less is known about why one setup is more effective
than another. In this work, we uncover the underlying characteristics of
effective sets. For each training instance, we form a large, diverse pool of
candidates and systematically vary the subsets used for calibration
fine-tuning. Each selection strategy targets distinct aspects of the sets, such
as lexical diversity or the size of the gap between positive and negatives. On
three diverse scientific long-form summarization datasets (spanning biomedical,
clinical, and chemical domains), we find, among others, that faithfulness
calibration is optimal when the negative sets are extractive and more likely to
be generated, whereas for relevance calibration, the metric margin between
candidates should be maximized and surprise--the disagreement between model and
metric defined candidate rankings--minimized. Code to create, select, and
optimize calibration sets is available at
https://github.com/griff4692/calibrating-summaries
The effectiveness of calibration setups for summarization models depends on the specific characteristics of the calibration sets, including lexical diversity and gap size between positive and negative sets.
MLE calibration step relevance faithfulness ranking calibration fine-tuning lexical diversity metric margin surprise