ZENY: Negative results are key to scientiﬁc progress
(Barwich, 2019), but are hard to establish if they
are very general. Published positive results often
overclaim their generality. Both should be con-
ﬁrmed only by accumulated evidence in diverse
settings.18

SOCART: No, no, you fail to see there’s a qualitative
difference, Zeny! Imagine if Ann was evaluating
self-attention for sentiment analysis. To answer
the hypothesis that self-attention works in the pos-
itive, she just needs a signiﬁcant result in a single
setting. In contrast, in order to establish a negative
result, she has to explore all possible settings.19
How would preregistration make establishing a
negative result less formidable a challenge?

ZENY: I agree that it would not. My only claim is that
evaluating studies prior to obtaining results would
prevent any bias on behalf of peer reviewers to
evaluate negative results more harshly.

SOCART:
A Two-Sided Discussion of Preregistration of NLP Research