In recent years, neural models have often outperformed rule-based and classic Machine Learning approaches in NLG. These classic approaches are now often disregarded, for example when new neural models are evaluated. We argue that they should not be overlooked, since for some tasks, well-designed non-neural approaches achieve better performance than neural ones. In this paper, the task of generating referring expressions in linguistic context is used as an example. We examined two very different English datasets ( WEBNLG and WSJ ), and evaluated each algorithm using both automatic and human evaluations. Overall, the results of these evaluations suggest that rule-based systems with simple rule sets achieve on-par or better performance on both datasets compared to state-of-the-art neural REG systems. In the case of the more realistic dataset, WSJ , a machine learning-based system with well-designed linguistic features performed best. We hope that our work can encourage researchers to consider non-neural models in future. Natural Language Generation (NLG) is concerned with the generation of natural language text from non-linguistic input (Gatt and Krahmer, 2018). One step in a classic generation pipeline (Reiter and Dale, 2000) is Referring Expression Generation (REG, Krahmer and van Deemter (2019) for an overview). REG has important practical value for commercial natural language generation (Re-iter, 2017), computer vision (Mao et al., 2016), and robotics (Fang et al., 2015), for example. It has also been used as a tool to understand human language use (van Deemter, 2016). REG contains two different problems. One is to find a set of attributes to single out a referent from a set (also  Equal contribution. called one-shot REG ). The other is to generate referring expressions (REs) to refer to a referent at different points in a discourse (Belz and Varges, 2007). We will focus on the latter task. We call this the REG-in-context task. In earlier works, REG is often tackled in two steps (Henschel et al., 2000; Krahmer and Theune, 2002). The first step decides the form of an RE. For example, whether a reference should be a proper name ( Marie Skodowska-Curie ), a description ( the physicist ), or a pronoun ( she ) at a given point in the context. The second step is concerned with content selection, i.e., the different ways in which a referential form can be realised. For example, to generate a description of Marie Curie , the REG system decides whether it is sufficient to mention her profession (i.e.,  the physicist ) or whether it is better to mention her nationality as well (i.e.,  a Polish-French physicist ). Thanks to the rapid development of deep learning techniques, recent NLG models are able to generate RE in an End2End (E2E) manner, i.e., to tackle the selection of form and content simultaneously (Castro Ferreira et al., 2018a; Cao and Cheung, 2019; Cunha et al., 2020). The task of End2End (E2E) REG was proposed by Castro Ferreira et al. (2018a), who extracted a corresponding corpus from the WebNLG corpus (Castro Ferreira et al., 2018b) 1 . Grounding on the WEBNLG dataset, they proposed a neural REG system built on a sequence-to-sequence with attention model. Their automatic and human evaluation results suggested that neural REG systems significantly outperform rule-based and feature-based machine learning (ML) baselines. However, it can be argued that Castro Ferreira et al. did not use very strong baselines for their comparison: OnlyName is a rule-based system that always generates a proper name given an entity, and Ferreira is a feature-based model 1 We refer to this extracted REG corpus as WEBNLG . that uses Naive Bayes with only 3 simple features . We present several rule-based and feature-based baselines to examine how neural models perform against well-designed non-neural alternatives. Note that a well-designed model is not necessarily complex. For example, it can be a rule-based system with one or two simple, well-designed rules. Since one of the advantages of neural E2E models is that they require little effort for feature engineering, we used two types of baselines, namely models that require minimal expert effort and models that use more demanding (but linguistically well-established) rules or features. Therefore, our main research question is: Do state-of-the-art neural REG models always perform better than rule-based and machine learning-based models? To answer this question fairly, we consider the amount of resources used by each model. For example, the neural models require fewer human resources when it comes to linguistic expertise and annotation, but they require input from Deep Learning experts. Resources such as computing power and data needs should also be considered. Another issue with previous studies concerns the datasets that were used: in WEBNLG , approximately 99.34% of entities in the test set also appear in the training set; consequently, evaluations using WEBNLG do not take unseen entities into consideration. Furthermore, since many sentences in WEBNLG are paraphrases of one another, evaluating neural models on WEBNLG alone may overestimate their performance. Castro Ferreira et al. (2019) recently extended WEBNLG to include unseen domains that contain many unseen entities 3 , and Cunha et al. (2020) have developed new models to handle them. Their test set has two subsets: one consists of documents 99.34% of whose entities are seen , while the other consists of documents 92.81% of whose entities are unseen . This arguably makes the data in WEBNLG unrealistic (see 2 for discussion). Therefore, we created what we believe to be a more realistic dataset based on the Wall Street Journal ( WSJ ) portion of the OntoNotes corpus (Hovy et al., 2006; Weischedel et al., 2013) 4 . 2 The human evaluation in Cunha et al. (2020) showed a slightly different result: the OnlyName model performed as well as the Neural REG models in terms of fluency, grammaticality, and adequacy. However, since their human evaluation involved only two subjects, these outcomes need to be approached with caution. 3 We used version 1.5 of the WEBNLG dataset in https: //github.com/ThiagoCF05/webnlg . Linguis-We evaluate all models on both WEBNLG and WSJ , using automatic and human evaluation experiments. The human experiments included a total of 240 participants and 16920 judgments. This paper is structured as follows: in 2 and 3, we describe the datasets used and the REG models. In 4, we provide a detailed description of our automatic and human evaluations. In 5 and 6, we compare the results across different dimensions and make suggestions for future studies. The code for reproducing the results in this article can be found at: https://github.com/ a-quei/neuralreg-re-evaluation . This section explains the REG-in-context task and the two English datasets used to conduct the experiments. Given a text whose REs have not yet been generated, and given the intended referent for each of these REs, the REG-in-context task is to build an algorithm that generates all these REs. Consider the delexicalised text in Table 1. Given the entity  AWH_Engineering_College , REG selects an RE based on the entity and its pre-context ( AWH_Engineering_College is in Kuttikkattoor , India in the state of Kerala . ), and its post-context ( has 250 employees and Kerala is ruled by Kochi . The Ganges River is also found in India. ). Gardent et al. (2017) introduced the WEBNLG corpus for evaluating NLG systems. Using crowd-sourcing, each crowdworker was asked to write a description for a given Resource Description Framework (RDF) triple (Table 1). The number of triples varied from 1 to 7. This corpus was later enriched and delexicalised (Castro Ferreira et al., 2018a,b) to fit the REG-in-context task. Castro Ferreira et al. (2019) further extended WEBNLG and divided the documents into test sets seen (where all data are from the same domains as the training data) and unseen (where all data are from different domains than the training data). This results in that almost all entities from the seen test set appear in the training set (9580 out of 9644), while only a tic Data Consortium (LDC) https://catalog.ldc. upenn.edu/LDC2013T19 . few entities from the unseen test set appear in the training set (688 out of 9644). Note that the maximum number of triples in the unseen set is five. So, one would expect the data in the unseen set to be less complex than the seen data. We used version 1.5 of WEBNLG , which contains 67,027, 8278, and 19,210 REs in the training, development, and test sets. From the point of view of the present study, WEBNLG has some notable shortcomings. For a start, it consists of rather formal texts that may not reflect the everyday use of REs, and in which very simple syntactic structures dominate. The texts in WEBNLG also stand out for other reasons. For example, the texts are extremely short, with an average length of only 1.4 sentences. Consequently, as many as 85% of the REs are first-mentions, while 71% of the REs are proper names. Finally, in any given test sample, either more than 90% of the entities are seen or more than 90% are unseen. Realistic data should contain a reasonable amount of mixtures of seen and unseen entities. For all these reasons, we decided to test all algorithms on a second corpus as well. Using the Wall Street Journal portion of the OntoNotes corpus, we constructed a new English REG dataset, following a similar approach as Castro Ferreira et al. (2018a). This corpus ( WSJ ) has very different characteristics from the WEBNLG . The WSJ consists of 582 newspaper articles containing 20,186, 2362 and 2781 REs in the training, development and test sets, respectively. The average length of the documents is 1189 words, and each document consists of 25 sentences on average. Furthermore, 23% of the instances are first-mention REs and the rest are subsequent mentions. For each RE, we created its preand post-context at the local sentence-level and added K preceding and following sentences to the local context. We refer to K as the context length and set K =2 for this experiment. To create the dataset, we first delexicalised the REs. The dataset contains nearly 8000 coreferential chains. The REs in each chain were replaced with corresponding delexicalised expressions (similar to table 1). For delexicalisation, we used (1) the POStag information, (2) the fine-grained annotation of the referential forms, and (3) the entity type of each referent. To delexi-calise human REs, for example, we looked for concise but informative REs such as the combination of first and last names (e.g., Barack Obama ) 5 . When such an expression was found in a coreferential chain, its delexicalised version (tokens being separated by underscores, e.g., Barack_Obama ) was assigned to all REs in the chain. We then moved on to the next tag. Below is the order in which the human referents were searched and delexicalised: [ firstname-lastname ] , [ title-firstname-lastname ] , [ modified firstname-lastname ] , [ title-lastname ] , [ lastname ] , [ modified-lastname ] , [ firstname ] . For more details on the preparation of the WSJ documents and a delexicalised example, see Appendix A. 3 REG Models In this section, we introduce the rule-based, ML-based, and the SOTA neural REG models. The term ML-based here refers to models that require feature engineering and follow a pipeline architecture. 5 The reason for choosing concise and informative REs for delexicalisation is that these labels are also used in the realisation process. Rule-based models have been widely used for generating REs in context (McCoy and Strube, 1999; Henschel et al., 2000). Here, we build rule-based systems for binary classification into two classes, namely pronominal and non-pronominal REs. Otherwise, r is realised as a non-pronominal RE. An entity r is defined as discourse-old if it has been mentioned in the previous context. A competitor is an entity that can be referred to with the same pronoun as r . We also build a dictionary that stores the pronouns associated with each entity. For seen entities, we extract pronouns from the training data. If an entity has multiple possible pronominal forms, we extract the most frequent one. For unseen entities, we determine their pronominal forms based on their meta-information, which is also used in E2E systems (Cunha et al., 2020). For example, if an entity in WEBNLG has the type PERSON and the gender FEMALE , we assign  she  to this entity. For the surface realisation of each entity, we realise its non-pronominal form by replacing the underscores in the entity label with whitespaces (e.g.,  Adenan_Satem  to  Adenan Satem ), as previously described by Castro Ferreira et al. (2018a). We realise the pronominal forms according to Castro Ferreira et al. (2016) by using the grammatical role of each entity (e.g., if the entity is in the object position, then we realise  he  as  him ). Linguistically-informed Rule-based System ( RREG-L ). We build RREG-L by adopting a set of pronominalisation rules from Henschel et al. (2000). The fundamental concepts used by these rules are the idea of local focus , which is a simpler implementation of the Centering Theory (Grosz et al., 1995), and parallelism , i.e., whether r and its antecedent in the previous sentence have the same grammatical role (Henschel et al., 2000). The RREG-L is described in detail in Appendix B. 3.2 ML-based REG The GREC Shared Task (Belz et al., 2010) triggered a plethora of ML-based models for building REG-in-context (e.g., Greenbacker and McCoy, 2009; Hendrickx et al., 2008). These models differ from each other in the features and the ML algorithms they have used. In this study, we build ML-based REG models using CatBoost (Prokhorenkova et al., 2018). It predicts whether a reference is realised as a pronoun, proper name, or description. Once the referential form is predicted, the next step is to select the content. The most frequent variant (with the same referential form as the predicted class) is selected in the training corpus given the referent and the full set of features. If no matching RE is found, a back-off method (Castro Ferreira et al., 2018a) is used, removing one feature at a time in order of importance. The order is calculated using the inherent feature importance method of the CatBoost algorithm. Depending on which features are used, we build two variants of ML-based models, namely ML-S and ML-L . The detailed list of the features used in these models can be found in Appendix C. Features obtained by minimum effort ( ML-S ). To find out what the upper bound is for a system that does not require any additional linguistic information or any additional annotation effort, we developed ML-S . In this model, we have relied only on the features that can be extracted directly from the corpus. Therefore, features such as grammatical role (which requires a syntactic parser) are not included in this model. Linguistically Informed Features ( ML-L ). To evaluate the upper bound performance of ML-based systems, we developed ML-L with the features that could affect the choice of referential form and could improve the overall accuracy of the REG systems suggested by the previous linguistic and computational studies (Ariel, 1990; Gundel et al., 1993; Brennan, 1995; Arnold and Griffin, 2007; Fukumura and Van Gompel, 2011; Kibrik et al., 2016; von Heusinger and Schumacher, 2019; Same and van Deemter, 2020). For example, we included features encoding grammatical role, recency, gender, and animacy in ML-L . Note that ML-L makes full use of the syntactic information 6 and entity meta-information (e.g., GENDER and TYPE which are also used by both the rule-based systems and the neural models). A limitation of the rule-based and ML-based models mentioned above is that they are not able to handle situations where an RE form (e.g., a proper name) can have multiple realisations, e.g., Lady Gaga/Stefani Germanotta. End2End NeuralREG can address this by generating REs from scratch. This study examines three NeuralREG systems that have been developed to deal with unseen entities as well. All of them were developed using the sequence-to-sequence with attention model (Bah-danau et al., 2014). ATT+Copy . Cunha et al. (2020) proposed using three bidirectional LSTMs (Hochreiter and Schmidhuber, 1997) to encode a pre-context, a post-context, and the proper name of an entity (i.e., replacing underscores in entity labels with whitespaces) into three hidden vectors h ( pre ) , h ( post ) and h ( r ) , respectively. An auto-regressive LSTM-based decoder generates REs based on context vectors. To handle unseen entities, Cunha et al. used the copy mechanism, which allows the decoder to copy words from the contexts directly as output. ATT+Meta . ATT+Meta (Cunha et al., 2020) used meta information of each entity to improve the quality of the generated REs. In each decoding step t , the context vector v ( c ) t is concatenated with meta information embeddings before being fed to the decoder. In WEBNLG , meta information are the entity type v ( type ) and gender embeddings v ( gender ) ; while in WSJ , in addition to v ( type ) and v ( gender ) , there is also plurality embedding v ( pl ) . ProfileREG . Cao and Cheung (2019) made ProfileREG to leverage the content of entity profiles extracted from Wikipedia. More specifically, instead of encoding the proper name of each entity, ProfileREG asks the entity encoder to encode the whole entity's profile to obtain h ( r ) . Note that since profiles of entities in WSJ are not accessible, we evaluate ProfileREG only on WEBNLG . We evaluated all the systems described in 3 on both WEBNLG and WSJ using automatic and human evaluations. We implemented the neural models based on the code of Cunha et al. (2020) and Cao and Cheung (2019) 7 . For WEBNLG , we used their original parameter setting, while for WSJ , we tuned the parameters on the development set and used the best parameter set. To determine the optimal context length K of WSJ , we varied K from 1 to 5 sentences before and after the target sentence, then tested ATT+Meta on the development set with the different K contexts. It reaches the best performance when K = 2 . Metrics. Following Cunha et al. (2020), we evaluated REG systems from 3 angles. (1) RE Accuracy and String Edit Distance (SED, Levenshtein, 1966) were used to evaluate the quality of each generated RE. (2) After adding the REs to the original document, BLEU (Papineni et al., 2002) and Text Accuracy were used to evaluate the output text. (3) Precision , recall , and F1 score were used to assess pronominalisation. Results of WEBNLG . Table 2 depicts the results of WEBNLG 8 . Overall, the classic ruleand ML-based models performed better than neural models, while neural models did a better job on pronominalisation. For generating REs, ML-L had the best performance, as it obtained the highest RE 7 ATT+Copy and ATT+Meta : github.com/ rossanacunha/NeuralREG ; and ProfileREG : github.com/mcao610/ProfileREG . 8 Note that there is a discrepancy between our replication results and the results of Cunha et al. (2020). The reason for this difference is that we found a bug in the code for preprocessing provided by the original paper and fixed it after consultation with Cunha et al. 5558 Model RE Acc. accuracy and BLEU scores and the second best SED and text accuracy score. For pronominalisation, ProfileREG yields the best performance, followed by RREG-S. We were surprised to find that the simplest rule-based system, RREG-S , performs remarkably well. It not only defeats the linguistically informed, rule-based RREG-L , but also outperforms the SOTA neural models ATT+Copy and ATT+Meta on both RE generation and pronominalisation. Table 3 shows the breakdown of the seen and unseen subsets. The SOTA neural models (i.e., ATT+Copy , ATT+Meta , and ProfileREG ) have the top 3 performance on seen data, and the worst RE generation performance (i.e., RE Acc., SED, BLEU, and Text Acc.) on unseen data. The ML-based models achieve the fourth and fifth best performance on seen data, and lower performance (but not as low as the neural models) on unseen data. The nature of WEBNLG could explain this drop in performance on unseen data: the models may have limited ability to handle unseen entities, for instance, because they fail to conduct domain transfer (remember that unseen data comes from different domains than seen data). Since rule-based systems do not rely on training data, this explanation does not apply to them, which explains why they did not show the same drop in performance. In fact, they performed even better on unseen data, possibly because unseen data contained fewer triples than seen data (see 2). Concretely, rule-based systems have lower REG accuracy but higher pronominalisation accuracy on unseen data compared to seen data. Additionally, ML-based models have low performance in the pronominalisation of unseen entities. The pronominalisation accuracy of the rule-based models is based on a 2-way distinction between a pronominal and a non-pronominal form, while the ML-based models make a 3-way distinction between a pronoun, a proper name and a description. Another factor that might have lowered the performance of the ML models is the annotation practices in WEBNLG . Since these models are data-driven, the quality of the annotations directly affects their performance. It appears that whenever a (nominal) RE starts with a determiner, it is marked in WEBNLG as description ; otherwise, it is marked as proper name . For instance, United States is marked as a proper name, while The United States is wrongly marked as a description. To allow comparison with previous work, we have not corrected the annotations, but it is important to keep in mind that this issue can cause ML-based models to underperform. Results of WSJ . Table 4 shows the results of WSJ . Once again, ML-L performs best both in RE generation and in pronominalisation, outperforming the other models by a large margin. RREG-L outperforms RREG-S on WSJ on all evaluation metrics, which could be seen as confirmation of our hunch that WSJ contains different, and potentially more naturalistic texts than WEBNLG (see 2.2). accuracy. Also, the inclusion of meta-information significantly boosts the recall of pronominalisation comparing ATT+Copy with ATT+Meta . Table 10 in Appendix D shows an original text and different outputs generated by the WSJ models. Materials. For WEBNLG seen entities, we randomly sampled 4 instances from each triple size group of 2-7 from the test set. In the case of the unseen data, we randomly chose 6 instances from size groups of 2-5. In this way, we obtained a total number of 48 reference instances (24 seen and 24 unseen). In addition to each reference instance, we selected its 7 different versions generated by the models (3 neural, 2 ML-based and 2 rule-based models). This yields a total of 384 items (48  8). Design. The 384 items were randomly distributed into 12 lists of 32 items. Each list was rated by 10 participants. Participants were asked to rate each text for its fluency (does the text flow in a natural, easy to read manner?), grammaticality (is the text grammatical (no spelling or grammatical errors)?) and clarity (does the text clearly express the data in the table?) on a 7-point Likert scale anchored by 1 (very bad) and 7 (very good). The definition of each criterion was taken from Castro Ferreira et al. (2018a). Participants. We used Amazon Mechanical Turk (MTurk) for human evaluation. We restricted MTurk workers to those located in the United States, with an approval rating of  95% and 1,000 or more HITs approved. We rejected workers if they: (1) gave human-produced descriptions a score lower than 2 more than 3 times; or (2) gave scores with a standard deviation less than 0.5. 120 workers (12 lists  10 workers) participated, providing us with 11520 judgements (384 items  3 criteria  10 judgements/item). The participants were 80 males, 36 females, and 4 oth-ers/unanswered, with an average age of 37. Results. Table 5 shows the results of the human evaluation WEBNLG . Few of the differences reach significance (using Wilcoxon's signed-rank test with Bonferroni correction 9 ), suggesting that WEBNLG may be ill-suited for differentiating between REG models 10 . The only two significant differences appear when comparing RREG-S with ATT+Meta and ProfileREG in terms of the grammaticality of unseen data. The results suggest that RREG-S is the best model for generating REs on WEBNLG , performing on a par with neural models on seen data and better than neural models on unseen data. Unlike our automatic evaluation, ATT+Meta does not outperform ATT+Copy in human evaluation. Materials. We randomly selected 30 documents from the test set of WSJ (reference text). We included the 6 different outputs generated by the 6 WSJ models (hereafter target texts). In this way we obtained a total of 180 reference-target pairs. Design. As mentioned in 2.3, the WSJ documents have an average length of 25 sentences. Since there are no input representations (e.g., in RDF) for WSJ , we decided to ask participants to compare texts using a Magnitude Estimation (ME) (Bard et al., 1996). The participants saw the reference and one of the target texts side by side, and they were asked to rate the target relative to the reference text. To make the task manageable for participants, texts were shortened to a maximum of the first 20 sentences. The 180 reference-target pairs were randomly distributed over 12 lists, each list having 15 items. Each list was rated by 10 participants. They were asked to rate the fluency, grammaticality and clarity of the target texts. The definition of fluency and grammaticality were as in the WEBNLG task, and clarity was defined as how clearly does the target text allow you to understand the situation described in the standard text 11 ?". The question asked for each of the 3 criteria was: assuming that standard text has a score of 100, how do you rate the fluency | grammaticality | clarity of target text? Participants were allowed to choose any positive number. Participants. The MTurk worker restrictions were similar to the WEBNLG experiment. Workers with scores less than 5 standard deviations were rejected. The experiment included 120 participants, resulting in 5400 judgements (180 items  3 criteria  10 judgements/item). The participants were 65 males, 54 females, and 1 oth-ers/unanswered, with an average age of 38. Results. Since typos are possible in ME (e.g., a worker might type 600 instead of 60), we excluded outliers, defined as a score that is lower than the median minus 3 standard deviations, or higher than the median plus 3 standard deviations of that item. The remaining scores were down-sampled for conducting significant testing. The results are shown in Table 6. Unlike WEBNLG , significant differences are frequent. For fluency, ML-S and ML-L perform the best while ATT+Meta performs the worst. For grammaticality, ML-L is still the best model, which significantly defeats RREG-L and ATT+Meta . A more detailed study is needed to investigate why RREG-L is the second worst in terms of grammaticality, which we found surprising. For clarity, no significant difference was found, perhaps because it was difficult for participants to compare long 11 We refer to the reference text as standard text . documents. In sum, on WSJ , ML-L has the best performance, and the simpler ML-S and RREG-S also have considerably good performances. Why does Neural REG not defeat rule-based REG? Received wisdom has it that although neural models may be inferior to other models in terms of interpretability, they are nonetheless superior in terms of performance. Although it is possible that future neural models will perform better than the ones examined here, our results call into question whether this received wisdom is correct. One possible explanation is the observation that Neural NLG systems tend to perform very well on surface realisation tasks, but less well on tasks that focus on semantic content (see e.g., Reiter (2018) on hallucinations in the Data2Text generation tasks). REG, after all, is a task that focuses in large part on semantic content. There may be other reasons, which should be investigated in future work. Role of Linguistically-informed Features. Rule-based models did particularly well on WEBNLG , outperforming other models. By contrast, on WSJ , the linguistically-informed feature-based model ( ML-L ) outperformed all other models. This suggests that the type of text, and consequently, the complexity of the REG task, might be a factor in choosing the REG method. Linguistically-informed features seem to have a more pivotal role in the case of more complex text types, whereas simpler texts can be handled at least as well by simpler rule-based models. Resources Use. As mentioned before, different approaches require different amounts of human resources and annotation efforts. But we believe that other resource types should also be taken into consideration when models are compared, including the following: (1) The amount of context : the neural models access the whole pre-context and post-context for WEBNLG , while they access K preceding and K following sentences around the target entity for WSJ . The ML-based models extract features taking only the current sentence and the whole pre-context into account. The rule-based models only look at the current sentence and the previous one; (2) External tools : the neural models need no external tools, while the rule-based and ML-L models need a syntactic parser (which is also used for constructing datasets); (3) 5561 External information : rule-based models, ML-L , and ATT+Meta need entities' meta-information. ProfileREG requires the profile description of each entity, which, for most REG tasks, is hard to obtain; (4) Computing resources : the neural models need GPUs while other models can be constructed using merely personal computers; (5) The amount of training data : the rule-based models need no training data, while other models require training data (large-scale naturalistic versions of which, for the task of REG, is not available). As we have seen, RREG-S and ML-S perform remarkably well on both WSJ and WEBNLG . Taking resources into consideration, the advantage of using a model such as RREG-S and ML-S becomes more pronounced. RREG-S uses less human resources, less context, less computing resources, and no training data 12 compared to other models. ML-S needs more context and training data; it probably also needs more human effort for feature engineering and selecting ML models, but it needs no external tools and no meta-information. In aggregate, one's choice of model may depend partially on what resources are available. For instance, for classic pipeline NLG systems, syntactic position and meta-information are often decided by earlier steps in the pipeline (Gatt and Krahmer, 2018). Therefore, if one's aim is to rapidly construct a pipeline NLG system, then RREG-S should probably be preferred. Generalisability. We used neural REG to illustrate the importance of non-neural baselines. Our findings may not be generalisable to End2End NLG. However, if complex rule/template-based NLG systems are taken into account, Duek et al. (2018) found that although these systems cannot defeat neural approaches, they still have competitive performance. It would be interesting to compare different types of models for other sub-tasks in the NLG pipeline (e.g., content determination, aggregation, and lexicalisation) in a similar way as has been done in the present paper 13 . In this work, we have re-evaluated state-of-the-art Neural REG systems by considering four well-12 well-12 The pronominal form (e.g., he or she) of an entity can either be extracted from the training data or decided by the entity's meta-information. 13 Note that pipelined NLG systems are sometimes thought to yield better outputs than fully End2End NLG systems (Castro Ferreira et al., 2019) designed ruleand ML-based baselines. In addition to the existing WEBNLG corpus, we built a new dataset for the task of REG-in-context on the basis of the WSJ corpus, arguing that this dataset may be more appropriate for the task. In the reevaluation, we examined both our baselines and SOTA neural REG systems on both datasets, using automatic and human evaluations. The results suggest that the simplest rule-based baseline RREG-S achieves equally good or better performance compared to SOTA neural models. Our results on the WSJ suggest that, on that corpus, the linguistically-informed ML-based model ( ML-L ) is best. We hope these results will encourage further research into the comparative strengths and weaknesses of neural, non-neural and hybrid methods in NLP. In future, we have 4 items on our TODO list: (1) Investigate bottleneck features for Neural based models based on the feature set of ML-L ; (2) Explore other neural architectures (e.g., testing models that leverage pre-trained language models) and construct larger realistic REG corpora; (3) Explore better human evaluation methods for longer documents that are better suited for evaluating the task of generating referring expressions in context; (4) Extend our research to other languages, especially in other language families, including languages that are morphological very rich or very poor and languages that frequently use zero pronouns (e.g., Chinese (Chen et al., 2018)). We thank the anonymous reviewers for their helpful comments. Guanyi Chen is supported by China Scholarship Council (No.201907720022). Fahime Same is supported by the German Research Foundation (DFG) Project-ID 281511265  SFB 1252 Prominence in Language and the Junior Special Fund of the Cologne Center of Language Sciences (CCLS). We collected our human evaluations using Amazon Mechanical Turk. For the WEBNLG task, which used a 7-point Likert scale, the workers were paid 0.03$ per item, in line with rates for similar tasks. For the more demanding WSJ task, we paid 0.10$ per item. The payment for each task was set at $7.5/hour (slightly above the US minimum wage, i.e., $7.25/hour). We expected the amount to be a fair remuneration, but given the actual time some 5562 participants needed, their remuneration turned out to be on the low side. In future crowd-sourcing experiments, we will base our remuneration on a more generous estimate of the duration per experimental task. We asked for demographic information, age, gender and English proficiency level, explicitly stating in the experiment that Your information will be used for research purposes only. All your data will be held anonymously ." These fields were not marked as mandatory fields. The demographic information will not be made publicly available.
abstain abstain result method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain method method result other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain objective result abstain result result other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain