Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. "Does My Rebuttal Matter? Insights from a Major NLP Conference." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1274-1290. 2019. The paper uses the term peer pressure to refer to the incentive to update the scores and reach a consensus. Such consensus is often explicitly encouraged by the area chairs, e.g. by enforcing a discussion for submissions with high deviation in scores while leaving submissions with low deviation intact. In our analysis, we use the data from the ACL 2018 conference to estimate which factor is more responsible for the score updates, the authors' rebuttal or the opinions of the other reviewers, expressed numerically as scores. For this, we conduct regression analyses. If the scores given by the other reviewers dominate in the regression, we take this as evidence for the peer pressure effect; if factors relating to the rebuttal dominate, we assume the peer pressure effect to be weaker. Our experiments in Section 4.3 show that peer pressure dominates the regression. Note that both the texts and the scores of the peer reviews can yield peer pressure. A reviewer can be influenced by the ideas expressed in the other reviewers' texts, or simply by the pressure to be aligned with the other reviewers in terms of scores. The latter kind of influence is also known as herd behavior (Banerjee, 1992) or conformity bias (Buechelet al., 2015). In our study, we do not distinguish the influence of these two factors, but instead treat the influence from the other peer reviews as a whole as peer pressure. Completely disentangling the causes of peer pressure would require controlled experimentation, e.g. allowing a group of reviewers to only see the scores while allowing another group to only see the comments, and analyze the score-update behavior differences in each group. Such experiments are to be set up in advance and integrated into the reviewing process itself and hence are beyond the scope of our study which operates on historical data. We encourage the community to perform such controlled experiments in the future.
abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method