Peer-review plays a critical role in the scientific writing and publication ecosystem. To assess the efficiency and efficacy of the reviewing process, one essential element is to understand and evaluate the reviews themselves. In this work, we study the content and structure of peer reviews under the argument mining framework, through automatically detecting (1) argumentative propositions put forward by reviewers, and (2) their types (e.g., evaluating the work or making suggestions for im-provement). We first collect 14 . 2 K reviews from major machine learning and natural language processing venues. 400 reviews are annotated with 10 , 386 propositions and corresponding types of EVALUATION , REQUEST , FACT , REFERENCE , or QUOTE . We then train state-of-the-art proposition segmentation and classification models on the data to evaluate their utilities and identify new challenges for this new domain, motivating future directions for argument mining. Further experiments show that proposition usage varies across venues in amount, type, and topic. Peer review is a process where domain experts scrutinize the quality of research work in their field, and it is a cornerstone of scientific discovery (Hettich and Pazzani, 2006; Kelly et al., 2014; Price and Flach, 2017). In 2015 alone, approximately 63 . 4 million hours were spent on peer reviews (Kovanis et al., 2016). To maximize their benefit to the scientific community, it is crucial to understand and evaluate the construction and limitation of reviews themselves. However, minimal work has been done to analyze reviews' content and structure, let alone to evaluate their qualities. As seen in Figure 1, peer reviews resemble arguments: they contain argumentative propositions (henceforth propositions) that convey reReview # 1 ( rating : 5 , # sentences : 11 ) [Quality: This paper demonstrates that convolutional and relational neural networks fail to solve visual relation problems ... ] FACT [This points at important limitations of current neural network architectures where architectures depend mainly on rote memorization. ] EVAL ... [Significance: This work demonstrates failures of relational networks on relational tasks ... ] FACT [Pros: Important message about network limitations. ] EVAL [Cons: Straightforward testing of network performance on specific visual relation tasks. ] EVAL ... Review # 2 ( rating : 5 , # sentences : 10 ) [The authors present two autoregressive models ... ] FACT ... [In that context , this work can be viewed as applying deep autoregressive density estimators to policy gradient methods. ] EVAL ... [At least one of those papers ought to be cited. ] REQ [It also seems like a simple, obvious baseline is missing from their experiments ... ] EVAL ... [The method could even be made to capture dependencies between different actions by adding a latent probabilistic layer ... ] EVAL ... [A direct comparison against one of the related methods in the discussion section would help ] REQ ... viewers' interpretation and evaluation of the research. Constructive reviews, e.g., review #2 , often contain in-depth analysis as well as concrete suggestions. As a result, automatically identifying propositions and their types would be useful to understand the composition of peer reviews. Therefore, we propose an argument mining-based approach to understand the content and structure of peer reviews . Argument mining studies the automatic detection of argumentative components and structure within discourse (Peld-szus and Stede, 2013). Specifically, argument types (e.g. evidence and reasoning) and their arrangement are indicative of argument quality (Habernal and Gurevych, 2016; Wachsmuth et al., 2017). In this work, we focus on two specific tasks: (1) proposition segmentation detecting elementary argumentative discourse units that are propositions, and (2) proposition classification  labeling the propositions according to their types (e.g., evaluation vs. request). Since there was no annotated dataset for peer reviews, as part of this study, we first collect 14 . 2 K reviews from major machine learning (ML) and natural language processing (NLP) venues. We create a dataset, AMPERE (Argument Mining for PEer REviews), by annotating 400 reviews with 10 , 386 propositions and labeling each proposition with the type of EVALUATION , REQUEST , FACT , REFERENCE , QUOTE , or NON-ARG . 1 Significant inter-annotator agreement is achieved for proposition segmentation (Cohen's  = 0 . 93 ), with good consensus level for type annotation (Krip-pendorf's  U = 0 . 61 ). We benchmark our new dataset with state-of-the-art and popular argument mining models to better understand the challenges posed in this new domain. We observe a significant drop of performance for proposition segmentation on AMPERE, mainly due to its different argument structure. For instance, 25% of the sentences contain more than one proposition, compared to that of 8% for essays (Stab and Gurevych, 2017), motivating new solutions for segmentation and classification. We further investigate review structure difference across venues based on proposition usage, and uncover several patterns. For instance, ACL reviews tend to contain more propositions than those in ML venues, especially with more requests but fewer facts. We further find that reviews with extreme ratings, i.e., strong reject or accept, tend to be shorter and make much fewer requests. Moreover, we probe the salient words for different proposition types. For example, ACL reviewers ask for more examples when making requests, while ICLR reviews contain more evaluation of network and how models are trained. We collect review data from three sources: (1) openreview.net an online peer reviewing platform for ICLR 2017, ICLR 2018, and UAI 2018 2 ; (2) reviews released for accepted papers at NeurIPS from 2013 to 2017; and (3) opted-in reviews for ACL 2017 from Kang et al. (2018). //xinyuhua.github.io/Resources/naacl19/ . 2 ICLR reviews are downloaded from the public API: https://github.com/iesl/openreview-py . UAI reviews are collected by the OpenReview team. EVALUATION : Subjective statements, often containing qualitative judgment. Ex: This paper shows nice results on a number of small tasks. REQUEST : Statements suggesting a course of action. Ex: The authors should compare with the following methods. FACT : Objective information of the paper or commonsense knowledge. Ex: Existing works on multi-task neural networks typically use hand-tuned weights ...  REFERENCE : Citations and URLs. Ex: see MuseGAN (Dong et al), MidiNet (Yang et al), etc  QUOTE : Quotations from the paper. Ex: The author wrote where r is lower bound of feature norm'. NON-ARG : Non-argumentative statements. Ex: Aha, now I understand. mining corpora, including # of annotated propositions. In total, 14 , 202 reviews are collected (ICLR: 4 , 057 ; UAI: 718 ; ACL: 275 ; and NeurIPS: 9 , 152 ). All venues except NeurIPS have paper rating scores attached to the reviews. Annotation Process. For proposition segmentation, we adopt the concepts from Park et al. (2015) and instruct the annotators to identify elementary argumentative discourse units on sentence or sub-sentence level, based on their discourse functions and topics. They then classify the propositions into five types with an additional non-argument category, as explained in Table 1. 400 ICLR 2018 reviews are sampled for annotation, with similar distributions of length and rating to those of the full dataset. Two annotators who are fluent English speakers first label the 400 reviews with proposition segments and types, and a third annotator then resolves disagreements. We calculate the inter-annotator agreement between the two annotators. A Cohen's  of 0 . 93 is achieved for proposition segmentation, with each review treated as a BIO sequence. For classification, unitized Krippendorf's  U (Krippendorff, 2004), which considers disagreements among segmentation, is calculated per review and then averaged over all samples, and the value is 0 . 61 . Among the exactly matched proposition segments, we report a Cohen's  of 0 . 64 . Statistics. Table 2 shows comparison between AMPERE and some other argument mining datasets of different genres. We also show the number of propositions in each category in Table 3. The most frequent types are evaluation ( 38 . 3% ) and fact ( 36 . 5% ). We benchmark AMPERE with popular and state-of-the-art models for proposition segmentation and classification. Both tasks can be treated as sequence tagging problems with the setup similar to Schulz et al. (2018). For experiments, 320 reviews ( 7 , 999 propositions) are used for training and 80 reviews ( 2 , 387 propositions) are used for testing. Following Niculae et al. (2017), 5 -fold cross validation on the training set is used for hyperparam-eter tuning. To improve the accuracy of tokeniza-tion, we manually replace mathematical formulas, variables, URL links, and formatted citation with special tokens such as <EQN> , <VAR> , <URL> , and <CIT> . Parameters, lexicons, and features used for the models are described in the supplementary material. We consider three baselines. FullSent : treating each sentence as a proposition. PDTB-conn : further segmenting sentences when any discourse connective (collected from Penn Discourse Treebank (Prasad et al., 2007)) is observed. RST-parser : segmenting discourse units by the RST parser in Feng and Hirst (2014). For learning-based methods, we start with Conditional Random Field ( CRF ) (Lafferty et al., 2001) with features proposed by Stab and Gurevych ((2017), Table 7), and BiLSTM-CRF , a bidirectional Long Short-Term Memory network (BiLSTM) connected to a CRF output layer and further enhanced with ELMo representation (Pe-ters et al., 2018). We adopt the BIO scheme for sequential tagging (Ramshaw and Marcus, 1999), with O corresponding to NON-ARG . Finally, we consider jointly modeling segmentation and classification by appending the proposition types to BI tags, e.g., B-fact, with CRF ( CRF-joint ) and BiLSTM-CRF ( BiLSTM-CRF-joint ). Table 4 shows that BiLSTM-CRF outperforms other methods in F1. More importantly, the perfor-Prec. mance on reviews is lower than those reached on existing datasets, e.g., an F1 of 86 . 7 is obtained by CRF for essays (Stab and Gurevych, 2017). This is mostly due to essays' better structure, with frequent use of discourse connectives. With given proposition segments, predicted or gold-standard, we experiment with proposition-level models to label proposition types. We utilize two baselines. Majority simply assigns the majority type in the training set. PropLexicon matches the following lexicons for different proposition types in order, and returns the first corresponding type with a match; if no lexicon is matched, the proposition is labeled as NONARG :  REFERENCE : <URL> , <CIT>  QUOTE : , , '  REQUEST : should, would be nice, why, please, would like to, need  EVALUATION : highly, very, unclear, clear, interesting, novel, well, important, similar, clearly, quite, good  FACT : author, authors, propose, present, method, parameters, example, dataset, same, incorrect, correct For supervised models, we employ linear SVM with a squared hinge loss and group Lasso regularizer (Yuan and Lin, 2006). It is trained with the top 500 features selected from Table 9 in (Stab and Gurevych, 2017) by  2 test. We also train a convolutional neural network ( CNN ) proposed by Kim (2014), with the same setup and pre-trained word embeddings from word2vec (Mikolov et al., 2013). Finally, results by joint models of CRF and BiLSTM-CRF are also reported. F1 scores for all propositions and each type are reported in Table 5. A prediction is correct when both segment and type are matched with the true labels. CNN performs better for types with significantly more training samples, i.e., evaluation and fact, indicating the effect of data size on neural model's performance. Joint models (CRF-joint and BiLSTM-CRF-joint) yield the best F1 scores for all categories when gold-standard segmentation is unavailable. Here we leverage the BiLSTM-CRF-joint model trained on the annotated AMPERE data to identify propositions and their types in unlabeled reviews from the four venues (ICLR, UAI, ACL, and NeurIPS), to understand the content and structure of peer reviews at a larger scale. Proposition Usage by Venue and Rating. Figure 2 shows the average number of propositions per review, grouped by venue and rating. Scores in 1  10 are scaled to 1  5 by (cid:100) x/ 2 (cid:101) , with 1 as strong reject and 5 as strong accept. ACL and NeurIPS have significantly more propositions than ICLR and UAI. Ratings, which reflect a reviewer's judgment of paper quality, also affect proposition usage. We find that reviews with extreme ratings, i.e., 1 and 5 , tend to have fewer propositions. We further study the distribution of proposition type in each venue. As observed in Figure 3, ACL Evaluation Request Fact Reference Quote Non-Arg 0 10 20 30 40 % of Proposition Type ICLR UAI ACL NeurIPS Figure 3: Distribution of proposition type per venue. reviews contain more requests but fewer facts than other venues. Specifically, we find that 94 . 6% of ACL reviews have at least one REQUEST proposition, compared to 81 . 5% for ICLR and 84 . 7% for UAI. We also show proposition type distribution based on ratings in Figure 4. Reviews with the highest rating tend to use fewer evaluation and reference, while reviews with ratings of 3  4 (bor-derline or weak accept) contain more requests. We further observe a sharp decrease of QUOTE usage in rating group 4 , and a surge of NON-ARG for rating group 5 , while FACT remains consistent across rating ranges. Proposition Structure. Argumentative structure, which is usually studied as support and attack relations, reveals how propositions are organized into coherent text. According to Park and Cardie (2018), 75% of support relations happen between adjacent propositions in user comments. We thus plot the proposition transition probability matrix in Figure 5, to show the argument structure in AMPERE. The high probabilities along the diagonal line imply that propositions of the same type are often constructed consecutively, with the exception of quote, which is more likely to be followed by evaluation. Proposition Type and Content. We also probe the salient words used for each proposition type, and the difference of their usage across venues. For each venue, we utilize log-likelihood ratio test (Lin and Hovy, 2000) to identify the represen-E VALUATIONREQUESTFACTREFERENCEQUOTE All Venues overall, unclear, not, contribution, seem, interesting please, could, should, if, why, would, more, suggest think, each, some, data, useful, written, proposes <URL> , et, al., conference, paper, proceedings, arxiv , paper, we, :, our ICLR network, general, acceptance, convinced, trained network, appendix, recommend, because, novelty training, results, work, then, image deep, ;, nips, tative words in each proposition type compared to other types. Table 6 shows both the commonly used salient words across venues and the unique words with top frequencies for each venue (  = 0 . 001 ,  2 test). For evaluation, all venues tend to focus on clarity and contribution, with ICLR discussing more about network and NeurIPS often mentioning equations. ACL reviews then frequently request for examples. There is a growing interest in understanding the content and assessing the quality of peer reviews. Authors' feedback such as satisfaction and helpfulness have been adopted as quality indicators (Latu and Everett, 2000; Hart-Davidson et al., 2010; Xiong and Litman, 2011). Nonetheless, they suffer from author subjectivity and are often influenced by acceptance decisions (Weber et al., 2002). Evaluation by experts or editors proves to be more reliable and informative (van Rooyen et al., 1999), but requires substantial work and knowledge of the field. Shallow linguistic features, e.g., sentiment words, are studied in Born-mann et al. (2012) for analyzing languages in peer reviews. To the best of our knowledge, our work is the first to understand the content and structure of peer reviews via argument usage. Our work is also in line with the growing body of research in argument mining (Teufel et al., 1999; Palau and Moens, 2009). Most of the work focuses on arguments in social media posts (Park and Cardie, 2014; Wei et al., 2016; Habernal and Gurevych, 2016), online debate portals or Oxford-style debates (Wachsmuth et al., 2017; Hua and Wang, 2017; Wang et al., 2017), and student essays (Persing and Ng, 2015; Ghosh et al., 2016). We study a new domain of peer reviews, and identify new challenges for existing models. We study the content and structure of peer reviews under the argument mining framework. AMPERE, a new dataset of peer reviews, is collected and annotated with propositions and their types. We benchmark AMPERE with state-of-the-art argument mining models for proposition segmentation and classification. We leverage the classifiers to analyze the proposition usage in reviews across ML and NLP venues, showing interesting patterns in proposition types and content. This research is based upon work supported in part by National Science Foundation through Grants IIS-1566382 and IIS-1813341. We are grateful to the OpenReview team, especially Michael Spector, for setting up the API to facilitate review data collection. We also thank three anonymous reviewers for their constructive suggestions.
abstain abstain method objective abstain abstain objective abstain abstain abstain abstain abstain abstain other method abstain abstain abstain objective abstain abstain method objective abstain method abstain objective result abstain objective abstain result abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain other other other other other objective abstain other objective method abstain abstain result other other other