CLIPPER: Compression enables long-context synthetic data generation
LLM developers are increasingly reliant on synthetic data, but generating
high-quality data for complex long-context reasoning tasks remains challenging.
We introduce CLIPPER, a compression-based approach for generating synthetic
data tailored to narrative claim verification - a task that requires reasoning
over a book to verify a given claim. Instead of generating claims directly from
the raw text of the book, which results in artifact-riddled claims, CLIPPER
first compresses the book into chapter outlines and book summaries and then
uses these intermediate representations to generate complex claims and
corresponding chain-of-thoughts. Compared to naive approaches, CLIPPER produces
claims that are more valid, grounded, and complex. Using CLIPPER, we construct
a dataset of 19K synthetic book claims paired with their source texts and
chain-of-thought reasoning, and use it to fine-tune three open-weight models.
Our best model achieves breakthrough results on narrative claim verification
(from 28% to 76% accuracy on our test set) and sets a new state-of-the-art for
sub-10B models on the NoCha leaderboard. Further analysis shows that our models
generate more detailed and grounded chain-of-thought reasoning while also
improving performance on other narrative understanding tasks (e.g.,
NarrativeQA).
67b7edf6a1d1394d1682c085
Chau Minh Pham
claimed_verified
2025-02-21T09:58:36.388Z
65b976fdf69f4d0377aef3fe
/avatars/1201194e2956c56b50098cc465a04c11.svg
Chau Minh Pham
user
chtmp223
67b7edf6a1d1394d1682c086
Yapei Chang
admin_assigned
2025-02-21T15:17:25.220Z
5f1dcc06cb8f993fa01f4775
https://cdn-avatars.huggingface.co/v1/production/uploads/1668660782327-5f1dcc06cb8f993fa01f4775.png
Yapei Chang
user
yapeichang
67b7edf6a1d1394d1682c087
Mohit Iyyer
admin_assigned
2025-02-21T15:17:30.925Z
6669df71d1652853e4b66ee5
/avatars/8a40bca9423ef76f39ae24d7a9e63478.svg
Mohit Iyyer
user
mohitiyyer
CLIPPER: Compression enables long-context synthetic data generation
LLM developers are increasingly reliant on synthetic data, but generating
high-quality data for complex long-context reasoning tasks remains challenging.
We introduce CLIPPER, a compression-based approach for generating synthetic
data tailored to narrative claim verification - a task that requires reasoning
over a book to verify a given claim. Instead of generating claims directly from
the raw text of the book, which results in artifact-riddled claims, CLIPPER
first compresses the book into chapter outlines and book summaries and then
uses these intermediate representations to generate complex claims and
corresponding chain-of-thoughts. Compared to naive approaches, CLIPPER produces
claims that are more valid, grounded, and complex. Using CLIPPER, we construct
a dataset of 19K synthetic book claims paired with their source texts and
chain-of-thought reasoning, and use it to fine-tune three open-weight models.
Our best model achieves breakthrough results on narrative claim verification
(from 28% to 76% accuracy on our test set) and sets a new state-of-the-art for
sub-10B models on the NoCha leaderboard. Further analysis shows that our models
generate more detailed and grounded chain-of-thought reasoning while also
improving performance on other narrative understanding tasks (e.g.,
NarrativeQA).
CLIPPER compresses book content into outlines and summaries to generate high-quality synthetic claims for narrative claim verification, improving model accuracy and performance.
CLIPPER synthetic data narrative claim verification chapter outlines book summaries chain-of-thoughts fine-tuning NoCha leaderboard NarrativeQA