We left one important thing in the open for now:
How do we fairly decide if a research subject and
protocol warrants low risk-tolerance? Ethics re-
view board members are already asked to ﬂag work
that ‘exhibits an increased risk of harm outside the
current norms of NLP or CL research’.27 This can
be hard to determine, but board members already
have to make this difﬁcult decision. Ethics reviews
could learn from established risk assessment frame-
works (Schwerdtner et al., 2020).

10 Conclusion

Our two-sided dialogue has discussed pros and
cons of preregistration in NLP, building on similar
discussions in epidemiology. What opponents else-
where have proposed as alternatives to preregistra-
tion is already found in NLP research: open access,
common repositories and data sheets (Lash and
Vandenbroucke, 2012). Preregistration, we argue,
is less urgently needed in ﬁelds that already facili-
tate replication and where risk of false positives

27https://aclrollingreview.org/
A Two-Sided Discussion of Preregistration of NLP Research