We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments. Even for such a particular alternation, several studies (Bresnan et al., 2007; Hovav and Levin, 2008; Colleman, 2009) investigated the factors determining this word order and found that the choice is not random. For analyzing such linguistic phenomena, linguists repeat the cycle of constructing hypotheses and testing their validity, usually through psychological experiments or count-based methods. However, these approaches sometimes face difficulties, such as scalability issues in psychological 0.0002 0.0000001 generationprobabilities order 1 is more likely. Compared to the typical approaches for evaluating linguistic hypotheses, approaches using LMs have potential advantages (Section 3.2). In this study, we examine the methodology of using LMs for analyzing word order (Figure 1). To validate the LM-based method, we first examine if there is a parallel between canonical word order and generation probability of LMs for each word order. Futrell and Levy (2019) reported that English LMs have human-like word order preferences, which can be one piece of evidence for validating the LM-based method. However, it is not clear whether the above assumption is valid even in languages with more flexible word order. In this study, we specifically focus on the Japanese language due to its complex and flexible word order. There are many claims on the canonical word order of Japanese, and it has attracted considerable attention from linguists and natural language processing (NLP) researchers for decades (Hoji, 1985; Saeki, 1998; Miyamoto, 2002; Matsuoka, 2003; Koizumi and Tamaoka, 2004; Nakamoto et al., 2006; Shigenaga, 2014; Sasano and Okumura, 2016; Orita, 2017; Asahara et al., 2018). We investigated the validity of using Japanese LMs for canonical word order analysis by conducting two sets of experiments: (i) comparing word order preference in LMs to that in Japanese speakers (Section 4), and (ii) checking the consistency Topic Time Location Subject (Adverb) Indirect object Direct object Verb Notation TOP TIM LOC NOMDAT ACC Typical particle between the preference of LMs with previous linguistic studies (Section 5). From our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool, and further explore potential applications. Finally, we analyzed the relationship between topicalization and word order of Japanese by taking advantage of the LM-based method (Section 6). Discuss and validate the use of LMs as a tool for word order analysis as well as investigate the sensitivity of LMs against different word orders in non-European language (Section 3);  Find encouraging parallels between the results obtained with the LM-based method and those with the previously established method on various hypotheses of canonical word order of Japanese (Sections 4 and 5); and  Showcase the advantages of an LM-based method through analyzing linguistic phenomena that is difficult to explore with the previous data-driven methods (Section 6). This section provides a brief overview of the linguistic background of canonical word order, some basics of Japanese grammar, and common methods of linguistic analysis. Every language is assumed to have a canonical word order, even those with flexible word order (Comrie, 1989). There has been a significant linguistic effort to reveal the factors determining the canonical word order (Bresnan et al., 2007; Hoji, 1985). The motivations for revealing the canonical word order range from linguistic interests to those involved in various other fieldsit relates to language acquisition and production in psycholinguistics (Slobin and Bever, 1982; Akhtar, 1999), second language education (Alonso Belmonte et al., 2000), and natural language generation (Visweswariah et al., 2011) or error correction (Cheng et al., 2014) in NLP. In Japanese, there are also many studies on its canonical word order (Hoji, 1985; Saeki, 1998; Koizumi and Tamaoka, 2004; Sasano and Okumura, 2016). Japanese canonical word order The word order of Japanese is basically subject-object-verb (SOV) order, but there is no strict rule except placing the verb at the end of the sentence (Tsujimura, 2013). For example, the following three sentences have the same denotational meaning ( A teacher gave a student a book. ): (2) a. .............. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) . teacherNOM studentDAT bookACC gave. b. .............. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) . teacherNOM bookACC studentDAT gave. c. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) .............. . bookACC studentDAT teacherNOM gave. This order-free nature suggests that the position of each constituent does not represent its semantic role (case). Instead, postpositional case particles indicate the roles. Table 1 shows typical constituents in a Japanese sentence, their postpositional particles, their canonical order, and the sections of this paper where each of them is analyzed. Note that postpositional case particles are sometimes omitted or replaced with other particles such as adverbial particles (Section 6). These characteristics complicate the factors determining word order, which renders the automatic analysis of Japanese word order difficult. There are two main methods in linguistic research: human-based methods, which observe human reactions, and data-driven methods, which analyze text corpora. Human-based methods A typical approach of testing word order hypotheses is observing the reaction (e.g., reading time) of humans to each word order (Shigenaga, 2014; Bahlmann et al., 2007). These approaches are based on the direct observation of humans, but this method has scalability issues. Data-driven methods Another typical approach is counting the occurrence frequencies of the targeted phenomena in a large corpus. This count-based method is based on the assumption that there are parallels between the canonical word order and the frequency of each word order in a large corpus. The parallel has been widely discussed (Arnon and Snider, 2010; Bresnan et al., 2007), and many studies rely on this assumption (Sasano and Okumura, 2016; Kempen and Harbusch, 2004). One of the advantages of this approach is suitability for large-scale experiments. This enables considering a large number of examples. In this method, researchers often have to identify the phenomena of interest with preprocessors (e.g., the predicate-argument structure parser used by Sasano and Okumura (2016)) in order to count them. However, sometimes, identification of the targeted phenomena is difficult for the preprocessors, which limits the possibilities of analysis. For example, Sasano and Okumura (2016) focused only on simple examples where case markers appear explicitly, and only extract the head noun of the argument to avoid preprocessor errors. Thus, they could not analyze the phenomena in which the above conditions were not met. The above issue becomes more serious in low-resource languages, where the necessary preprocessors are often unavailable. In this count-based direction, Bloem (2016) used n-gram LMs to test the claims on the German two-verb clusters. This method is closest to our proposed approach, but the general validity of using LMs is out of focus. This LM-based method also relies on the assumption of the parallels between the canonical word order and the frequency. Another common data-driven approach is to train an interpretable model (e.g., Bayesian linear mixed models) to predict the targeted linguistic phenomena and analyze the inner workings of the model (e.g., slope parameters) (Bresnan et al., 2007; Asahara et al., 2018). Through this approach, researchers can obtain richer statistics, such as the strength of each factor's effect on the targeted phenomena, but creating labeled data and designing features for supervised learning can be costly. In the NLP field, LMs are widely used to estimate the acceptability of text (Olteanu et al., 2006; Kann et al., 2018). An overview of the LM-based method is shown in Figure 1. After preparing several word orders considering the targeted linguistic hypothesis, we compare their generation probabilities in LMs. We assume that the word order with the highest generation probability follows their canonical word order. In the count-based methods mentioned in Section 2.2, researchers often require preprocessors to identify the occurrence of the phenomena of interest in a large corpus. On the other hand, researchers need to prepare data to be scored by LMs to evaluate hypothesis in the LM-based method. Whether it is easier to prepare the preprocessor or the evaluation data depends on the situation. For example, the data preparation is easier in the situation where one wants to analyze the word order trends when a specific postpositional particle is omitted. The question is whether Japanese speakers prefer the word order like in Example (3)-a or (3)-b. 1 While identifying the cases ( ACC in Example (3)) without their postpositional particle is difficult, creating the data without a specific postpositional particle by modifying the existing data is easier such as creating Example (4)-b from Example (4)-a. The human-based method is more reliable given an example. However, it can be prohibitively costly. While the human-based method requires an evaluation data and human subjects, the LM-based method only requires the evaluation data. Thus, the LM-based method can be more suitable for estimating the validity of hypotheses and considering 1 Omitted characters are crossed out. (e.g.,  ) many examples as exhaustively as possible. In addition, the LM-based method can be replicable. The suitable approach can be different in a situation, and broadening the choice of alternative methodologies may be beneficial to linguistic research. Nowadays, various useful frameworks, language resources, and machine resources required to train LMs are available, 2 which support the ease of implementing the LM-based method. Moreover, we make the LMs used in this study available. 3 3.3 Strategies to validate the use of LMs to analyze the word order The goal of this study is to validate the use of LMs for analyzing the canonical word order. The canonical word order itself is still a subject of research, and the community does not know all about it. Thus, it is ultimately impossible to enumerate the requirements on what LMs should know about the canonical word order and probe the knowledge of LMs. Instead, we demonstrate the validity of the LM-based method by showcasing two types of parallels: (i) word order preference of LMs showing parallels with that of humans, and (ii) the results obtained with the LM-based method and those with previous methods being consistent on various claims on canonical word order. If the results of LMs are consistent with those of existing methods, the possibility that LMs and existing methods have the same ability to evaluate the hypotheses is supported. If the LM-based method is assumed to be valid, the method has the potential to streamline the research on unevaluated claims on word order. In the experiment sections, we examine the properties of Japanese LMs on (i) and (ii). Even if LMs satisfy the criteria described in 3.3, there is no exact guarantee that LM scores will re-flect the effectiveness of human processing of specific constructions in general. Thus, there seems to be a danger of confusing LM artifacts with language facts. Based on this, we hope that researchers use LMs as a tool just to limit the hypothesis space. LM supported hypotheses should then be re-verified with a human-based approach. Furthermore, since there is a lot of hypotheses and corresponding research, we cannot check all the properties of LMs in this study. This study focuses on intra-sentential factors of Japanese case order, and it is still unclear whether the LM-based method works properly in linguistic phenomena which are far from being the focus of this study. This is the first study where evidence is collected on the validity of using LMs for word order analysis and encourages further research on collecting such evidence and examining under what conditions this validity is guaranteed. We used auto-regressive, unidirectional LMs with Transformer (Vaswani et al., 2017). We used two variants of LMs, a character-based LM (CLM) and a subword-based LM (SLM). In training SLM, the input sentences are once divided into morphemes by MeCab (Kudo, 2006) with a UniDic dictionary, 4 and then these morphemes are split into subword units by byte-pair-encoding. (Sennrich et al., 2016) 5 . 160M sentences 6 randomly selected from 3B web pages were used to train the LMs. Hyperparameters are shown in Appendix A. Given a sentence s , we calculate its generation probability p ( s ) =  p ( s ) p ( s ) , where  p (  ) and  p (  ) are generation probabilities calculated by a left-to-right LM and a right-to-left LM, respectively. Depending on the hypothesis, we compare the generation probabilities of various variants of s with different word orders. We assume that the word order with the highest generation probability follows their canonical word order. To examine the validity of using LMs for canonical word order analysis, we examined the parallels between the LMs and humans on the task determining the canonicality of the word order (Figure 2). First, we created data for this task (Section 4.1). We then compared the word order preference of LMs and that of humans (Section 4.2). 4 https://unidic.ninjal.ac.jp/ 5 Implemented in sentencepiece (Kudo and Richardson, 2018) We set character coverage to 0.9995  and vocab size to 100,000. 6 14GB in UTF-8 encoding. For reference, Japanese Wikipedia has around 2.5 GB of text. Because the focus of this study has context-independent nature, the sentences order is shuffled to prevent learning the inter-sentential characteristics of the language. Data We randomly collected 10k sentences from 3B web pages, which are not overlapped with the LM training data. To remove overly complex sentences, we extracted sentences that must: (i) have less than or equal to five clauses and one verb, (ii) have clauses with a sibling relationship in its dependency tree, and they accompany a particle or adverb, (iii) not have special symbols such as parentheses, and (iv) not have a backward dependency path. For each sentence, we created its scrambled version. 7 The scrambling process is as follows: 1. Identify the dependency structure by using JUMAN 8 and KNP 9 . 2. Randomly select a clause with several children. 3. Shuffle the position of its children along with their descendants. Annotation We used the crowdsourcing platform Yahoo Japan! 10 . For our task, we showed crowdworkers a pair of sentences (order 1 , order 2 ), where one sentence has the original word order, and the other sentence has a scrambled word order. 11 Each annotator was instructed to label the pair with one of the following choices: (1) order 1 is better, (2) order 2 is better, or (3) the pair contains a semantically broken sentence. Only the sentences (order 1 , order 2 ) were shown to the annotators, and they were instructed not to imagine a specific context for the sentences. We filtered unmotivated workers by using check questions. 12 For each pair 7 When several scrambled versions were possible for a given sentence, we randomly selected one of them. 8 http://nlp.ist.i.kyoto-u.ac.jp/EN/ index.php?JUMAN 9 http://nlp.ist.i.kyoto-u.ac.jp/EN/ index.php?KNP 10 https://crowdsourcing.yahoo.co.jp/ 11 Crowdworkers did not know which sentence was the original sentence. 12 We manually created check questions considering the Japanese speakers' preference in trial experiments in advance. instance, we employed 10 crowdworkers. In total, 756 unique, motivated crowdworkers participated in our task. From the annotated data, we collected only the pairs satisfying the following conditions for our experiments: (i) none of 10 annotators determined that the pair contains a semantically broken sentence, and (ii) nine or more annotators preferred the same order. The majority decision is labeled in each pair; the task is binary classification. We assume that if many workers prefer a certain word order, then it follows its canonical word order, and the other one deviates from it. We collected 2.6k pair instances of sentences. We compared the word order preference of LMs and that of the workers by using the 2.6K pairs created in Section 4.1. We calculated the correlation of the decisions between the LMs and the workers; which word order is more appropriate order 1 or order 2 . The word orders supported by CLM and SLM are highly correlated with workers, with the Pearson correlation coefficient of 0.89 and 0.90, respectively. This supports the assumption that the generation probability of LMs can determine the canonical word order as accurately as humans do. Note that such a direct comparison of word order is difficult with the count-based methods because of the sparsity of the corpus. This section examines whether LMs show word order preference consistent with previous linguistic studies. The results are entirely consistent, which support the validity of the LM-based methods in Japanese. Each subsection focuses on a specific component of Japanese sentences. The order of double objects is one of the most controversial topics in Japanese word order. Examples of the possible order are as follows: Henceforth, DAT-ACC / ACC-DAT denotes the word order in which the DAT / ACC argument precedes the ACC / DAT argument. We evaluate the 0 0.5 1 0 0.5 1 ACC-DAT r a t e (c) Relationship between the degree of co-occurrence of verb and arguments, and the ACC-DAT rate in each example. For the results of LMs, the ACC-DAT rate of each example is regarded as 1 if LMs prefer ACC-DAT order, otherwise we regard the example as 0. Figure 3: Overlap of the results of Sasano and Okumura claims Sasano and Okumura (2016) focused on with the data they collected. 13 Word order for each verb First, we analyzed the trend of the double object order for each verb. We analyzed 620 verbs following Sasano and Okumura (2016). 14 For each set of examples S v corresponding to a verb v , we: (i) created an instance with the swapped order of ACC and DAT for each example, and (ii) compared the generation probabilities of the original and swapped instance. S v is the set of examples preferred by LMs. R v ACC-DAT is calculated as follows: R v ACC-DAT = N v ACC-DATN v ACC-DAT + N v DAT-ACC , where N v ACC-DAT / N v DAT-ACC is the number of examples with the ACC-DAT / DAT-ACC order in  S v . Figure 3-(a) shows the relationship between R v ACC-DAT determined by LMs and one reported in a 13 We filtered the examples overlapping with the training data of LMs in advance. previous count-based study (Sasano and Okumura, 2016). These results strongly correlate with the Pearson correlation coefficient of 0.91 and 0.88, in CLM and SLM, respectively. In addition, canon-ical word order is DAT-ACC (Hoji, 1985) is unlikely to be valid because there are verbs where R v ACC-DAT is very high (details in Appendix B.1). This conclusion is consistent with Sasano and Okumura (2016). Word order and verb types In Japanese, there are show-type and pass-type verbs (details in Appendix B.2). Matsuoka (2003) claimed that the order of double objects differs depending on these verb types. Following Sasano and Okumura (2016), we analyzed this trends. We applied the Wilcoxon rank-sum test between the distributions of R v ACC-DAT determined by LMs in the two groups (show-type and pass-type verbs). The results show no significant difference between the two groups (p-value is 0.17 and 0.12 in the experiments using CLM and SLM, respectively). These results are consistent with the count-based (Sasano and Okumura, 2016) and the human-based (Miyamoto, 2002; Koizumi and Tamaoka, 2004) methods. Word order and argument omission Sasano and Okumura (2016) claimed that the frequently omitted case is placed near the verb. First, we calculated R v DAT -only for each verb v as follows: R v DAT -only = N v DAT -only N v DAT -only + N v ACC -only , where N v DAT -only / N v ACC -only denotes the number of examples in which the DAT / ACC case appears, and the other case does not in S v . A large R v DAT -only score indicates that the DAT argument is less frequently omitted than the ACC argument in S v . We analyzed the relationship between R v DAT -only and R v ACC-DAT for each verb. Figure 3-(b) shows that the regression lines from the LM-based method and Sasano and Okumura (2016) corroborate similar trends. The Pearson correlation coefficient between R v DAT -only and R v ACC-DAT is 0.404 for CLM and 0.374 for SLM. The results are consistent with Sasano and Okumura (2016), where they reported that the correlation coefficient was 0.391. Word order and semantic role of the dative argument Matsuoka (2003) claimed that the canonical word order differs depending on the semantic role of the dative argument. Sasano and Okumura TIM < LOC TIM < NOM LOC < NOMCLM .757 .642 .604 SLM .708 .632 .615 Count .686 .666 .681 Table 2: The columns a < b show the score o Type-A has an inanimate goal ( school ) as the DAT argument, while Type-B has an animate processor ( teacher ). It was reported that Type-A is likely to be the ACC-DAT order, while Type-B is likely to be the DAT-ACC order. Following Sasano and Okumura (2016), we analyzed 113 verbs. 15 For each verb, we compared the ACC-DAT rate in its type-A examples and the rate in its type-B examples. The number of verbs where the ACC-DAT order is preferred in Type-A examples to Type-B examples is significantly larger (a two-sided sign test p < 0.05). This result is consistent with that of Sasano and Okumura (2016); Matsuoka (2003) and implies that the LMs capture the animacy of the nouns. Details are in Appendix B.3. Word order and co-occurrence of verb and arguments Sasano and Okumura (2016) claimed that an argument that frequently co-occurs with the verb tends to be placed near the verb. For each example, the LMs determine which word order ( DAT-ACC or ACC-DAT ) is appropriate. Each example also has a score  NPMI (definition in Appendix B.4). Higher  NPMI means that the DAT noun in the example more strongly co-occurs with the verb in the example than the ACC noun. Figure 3-(c) shows the relationship between  NPMI and the ACC-DAT rate in each example. NPMI and the ACC-DAT rate are correlated with the Pearson correlation coefficient of 0.517 and 0.521 in CLM and SLM, respectively. These results are consistent with Sasano and Okumura (2016). 15 Among the 126 verbs used in Sasano and Okumura (2016), 113 verbs with data that do not overlap with the LM training data were selected. Our focus moves to the cases closer to the beginning of the sentences. The following claim is a well-known property of Japanese word order: The case representing time information ( TIM ) is placed before the case representing location information ( LOC ), and the TIM and LOC cases are placed before the NOM case (Saeki, 1960, 1998). We examined a parallel between the result obtained with the LM-based and count-based methods on this claim. We randomly collected 81k examples from 3B web pages. 16 To create the examples, we identified the case components by KNP, and the TIM and LOC cases were categorized with JUMAN (details in Appendix C). For each example s , we created all possible word orders and obtained the word order with the highest generation probability (  s ). Given  S a set of  s , we calculated a score o ( a < b ) for cases a and b as follows: o where N k<l is the number of examples where the case k precedes the case l in  S . Higher o ( a < b ) indicates that the case a is more likely to be placed before the case b . The results with the LM-based methods and the count-based method are consistent (Table 2). Both results show that o ( TIM < LOC ) is significantly larger than o ( TIM > LOC ) ( p < 0 . 05 with a two-sided signed test), which indicates that the TIM case usually precedes the LOC case. Similarly, the results indicate that the TIM case and the LOC case precedes the NOM case. We checked the preference of the adverb position in LMs. The position of the adverb has no restriction except that it must be before the verb, which is similar to the trend of the case position. However, Koizumi and Tamaoka (2006) claimed that There is a canonical position of an adverb depend-16 Without overlap with the training data of LMs. Model long precedes short short precedes long CLM 5,640 3,754 SLM 5,757 3,914 Table 4: Changes in the position of a constituent with the largest number of chunks. ing on its type. adverbs: MODAL , TIME , MANNER , and RESULTIVE . We used the same examples as Koizumi and Tamaoka (2006). For each example s , we created its three variants with a different adverb position as follows ( A friend handled the tools roughly. ): (10) ASOV:    roughly    friendNOM    toolsACC   handled. SAOV:    friendNOM    roughly    toolsACC   handled. SOAV:    friendNOM    toolsACC    roughly   handled. where the sequence of the alphabet such as ASOV denote the word order of its corresponding sentences. For example, ASOV indicates the order: adverb < subject < object < verb. A, S, O, and V denote adverb, subject, object, and verb, respectively. Then, we obtained the preferred adverb position by comparing their generation probabilities. Finally, for each adverb type and its examples, we ranked the preference of the possible adverb positions: ASOV, SAOV, and SOAV. Table 3 shows the rank correlation of the preference of the position of each adverb type. The results show similar trends of LMs with that of the human-based method (Koizumi and Tamaoka, 2006). The effects of long-before-short, the trend that a long constituent precedes a short one, has been reported in several studies (Asahara et al., 2018; Orita, 2017) We checked whether this effect can be captured with the LM-based method. Among the examples used in Section 5.2, we analyzed about 9.5k examples in which the position of the constituent with the largest number of chunks 17 differed between its canonical case order 18 and the order supported by LMs. Table 4 shows that there are significantly (p < 0.05 with a two-sided signed test) large numbers 17 chunks were identified by KNP. of examples where the longest constituent moves closer to the beginning of the sentence. This result is consistent with existing studies and supports the tendency for longer constituents to appear before shorter ones. We found parallels between the results with the LM-based method and that with the previously established method on various properties of canonical word order. These results support the use of LMs for analyzing Japanese canonical word order. In the previous section, we tentatively concluded that LMs can be used for analyzing the intra-sentential properties on the canonical word order. Based on this finding, in this section, we demonstrate the analysis of additional claims on the properties of the canonical word order with the LM-based method, which has been less explored by large-scale experiments. This section shows the analysis of the relationship between topicalization and the canonical word order. Additional analyses on the effect of various adverbial particles for the word order are shown in Appendix F. 6.1 Topicalization in Japanese The adverbial particle ( TOP ) is usually used as a postpositional particle when a specific constituent represents the topic of the sentence (Hey-cock, 1993; Noda, 1996; Fry, 2003). When a case component is topicalized, the constituent moves to the beginning of the sentence, and the particle ( TOP ) is added (Noda, 1996). Additionally, the original case particle is sometimes omitted, 19 which makes the case of the constituent difficult to identify. For example, to topicalize ( book ACC ) in Example (8)-a, the constituent moves to the beginning of the sentence, and the original accusative case particle ( ACC ) is omitted. Similarly, ( teacher NOM ) is topicalized in Example (8)-b. The original sentence is enclosed in the square brackets in Example (8). 19 The particles ( ACC ) and ( NOM ) are omitted. With the above process, we can easily create a sentence with a topicalized constituent. On the other hand, identifying the original case of the topicalized case components is error-prone. Thus, the LM-based method can be suitable for empirically evaluating the claims related to the topicalization. By using the LM-based method, we evaluate the following two claims: (i) The more anterior the case is in the canonical word order, the more likely its component is topicalized (Noda, 1996). (ii) The more the verb prefers the ACC-DAT order, the more likely the ACC case is topicalized than the DAT case. The claim (i) suggests that, for example, the NOM case is more likely to be topicalized than the ACC case because the NOM case is before the ACC case in the canonical word order of Japanese. The claim (ii) is based on our observation. It can be regarded as an extension of the claim (i) considering the effect of the verb on its argument order. We assume that the canonical word order of Japanese is TIM < LOC < NOM < DAT < ACC in this section. Claim (i) We examine which case is more likely to be topicalized. We collected 81k examples from Japanese Wikipedia (Details are in Appendix C). For each example, a set of candidates was created by topicalizing each case, as shown in Example (8). Then, we selected the sentences with the highest score by LMs in each candidate set. We denote the obtained sentences as  S topic . We calculated a score t a | b for pairs of cases a and b . where N a | b is the examples where the case a and b appear, and case a is a topic of the sentence in  S topic . The higher the score is, the more the case a is likely to be topicalized than the case b is. We compared t a | b and t b | a among the pairs of cases a and b , where the case a precedes the case b in the canonical word order. Through our experiments, t a | b was significantly larger than t b | a ( p < 0 . 05 with a paired t-test) in CLM and SLM results, which supports the claim (i) (Noda, 1996). Detailed results are shown in Appendix E. Claim (ii) The canonical word order of double objects is different for each verb (Section 5.1). Based on this assumption and the claim (i), we hypothesized that the more the verb prefers the ACC-DAT order, the more likely the ACC case of the verb is topicalized than the DAT case. We used the same data as in Section 5.1. For each example, we created two sentences by topicalizing the ACC or DAT argument. Then we compared their generation probabilities. In each set of examples corresponding to a verb v , we calculated the rate that the sentence with the topicalized ACC argument is preferred rather than that with the topicalized DAT argument. This rate and R v ACC-DAT is significantly correlated with the Pearson correlation coefficient of 0.89 and 0.84 in CLM and SLM, respectively. This results support the claim (ii). Detailed results are shown in Appendix E. 7 Conclusion and Future work We have proposed to use LMs as a tool for analyzing word order in Japanese. Our experimental results support the validity of using Japanese LMs for canonical word order analysis, which has the potential to broaden the possibilities of linguistic research. From an engineering view, this study supports the use of LMs for scoring Japanese word order automatically. From the viewpoint of the linguistic field, we provide additional empirical evidence to various word order hypotheses as well as demonstrate the validity of the LM-based method. We plan to further explore the capability of LMs on other linguistic phenomena related to word order, such as given new ordering (Nak-agawa, 2016; Asahara et al., 2018). Since LMs are language-agnostic, analyzing word order in another language with the LM-based method would also be an interesting direction to investigate. Furthermore, we would like to extend a comparison between machine and human language processing beyond the perspective of word order. We would like to offer our gratitude to Kaori Uchiyama for taking the time to discuss our paper and Ana Brassard for her sharp feedback on English. We also would like to show our appreciation to the Tohoku NLP lab members for their valuable advice. We are particularly grateful to Ryohei Sasano for sharing the data for double objects order analyses. This work was supported by JSTCREST Grant Number JPMJCR1513, JSPS KAK-ENHI Grant Number JP19H04162, and Grant-in-Aid for JSPS Fellows Grant Number JP20J22697.
method abstain objective method abstain abstain abstain abstain objective abstain abstain abstain abstain method objective abstain abstain method other objective abstain abstain abstain abstain objective method abstain objective objective objective objective other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other objective other other other other other other other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain other other method method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other