LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Peer review is a cornerstone of quality control in scientific publishing.
With the increasing workload, the unintended use of `quick' heuristics,
referred to as lazy thinking, has emerged as a recurring issue compromising
review quality. Automated methods to detect such heuristics can help improve
the peer-reviewing process. However, there is limited NLP research on this
issue, and no real-world dataset exists to support the development of detection
tools. This work introduces LazyReview, a dataset of peer-review sentences
annotated with fine-grained lazy thinking categories. Our analysis reveals that
Large Language Models (LLMs) struggle to detect these instances in a zero-shot
setting. However, instruction-based fine-tuning on our dataset significantly
boosts performance by 10-20 performance points, highlighting the importance of
high-quality training data. Furthermore, a controlled experiment demonstrates
that reviews revised with lazy thinking feedback are more comprehensive and
actionable than those written without such feedback. We will release our
dataset and the enhanced guidelines that can be used to train junior reviewers
in the community. (Code available here:
https://github.com/UKPLab/arxiv2025-lazy-review)
67ff9068de173bffef0e7eae
Sukannya Purkayastha
claimed_verified
2025-04-16T11:57:51.596Z
6408e9f2bf3e9a4bb2e09400
https://cdn-avatars.huggingface.co/v1/production/uploads/1678306212334-6408e9f2bf3e9a4bb2e09400.jpeg
Sukannya Purkayastha
user
sukannya
67ff9068de173bffef0e7eaf
Zhuang Li
67ff9068de173bffef0e7eb0
Anne Lauscher
67ff9068de173bffef0e7eb1
Lizhen Qu
67ff9068de173bffef0e7eb2
Iryna Gurevych
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Peer review is a cornerstone of quality control in scientific publishing.
With the increasing workload, the unintended use of `quick' heuristics,
referred to as lazy thinking, has emerged as a recurring issue compromising
review quality. Automated methods to detect such heuristics can help improve
the peer-reviewing process. However, there is limited NLP research on this
issue, and no real-world dataset exists to support the development of detection
tools. This work introduces LazyReview, a dataset of peer-review sentences
annotated with fine-grained lazy thinking categories. Our analysis reveals that
Large Language Models (LLMs) struggle to detect these instances in a zero-shot
setting. However, instruction-based fine-tuning on our dataset significantly
boosts performance by 10-20 performance points, highlighting the importance of
high-quality training data. Furthermore, a controlled experiment demonstrates
that reviews revised with lazy thinking feedback are more comprehensive and
actionable than those written without such feedback. We will release our
dataset and the enhanced guidelines that can be used to train junior reviewers
in the community. (Code available here:
https://github.com/UKPLab/arxiv2025-lazy-review)
A dataset called LazyReview is introduced to improve the detection of lazy thinking in peer reviews, showing that instruction-based fine-tuning of LLMs enhances their ability to identify such heuristics.
peer-review sentences lazy thinking LazyReview Large Language Models zero-shot setting instruction-based fine-tuning peer reviews