ask how afraid we should be of being wrong. In
clinical trials, there is a signiﬁcant cost to being
wrong. In biomedical studies, false positive rates
have been found to be around 14% (Jager and Leek,
2014). Whatever the number is for NLP, lowering
it by adding more checks, will lead to a drop in the
true positive rate. If a false positive could result in
human tragedy, the price of a lower true positive
rate is worth paying, but in NLP, the cost of a false
positive is often paid for in compute and human
hours. While both can be scarce resources, the
open access nature of NLP makes being wrong less
dangerous, since mistakes are quickly corrected.24
Zeny would object that pretraining of language
models is not easily reproducible (Bender et al.,
2021). Pretraining very large language models
should maybe be required to preregister and this
would possibly require revising the questionnaires
provided by van Miltenburg et al. (2021). Another
concern is how NLP contributes to social and cul-
A Two-Sided Discussion of Preregistration of NLP Research