The learning trajectories of linguistic phenomena in humans provide insight into linguistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker. To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different models are similar enough in the generalizations they make. In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena. Taking inspiration from psycholinguistics, we argue that studying this inductive bias is an opportunity to study the linguistic representation implicit in NLMs. Leveraging these findings, we compare the relative performance on different phenomena at varying learning stages with simpler reference models. Results suggest that NLMs exhibit consistent developmental stages. Moreover, we find the learning trajectory to be approximately one-dimensional: given an NLM with a certain overall performance, it is possible to predict what linguistic generalizations it has already acquired. Initial analysis of these stages presents phenomena clusters (notably morphological ones), whose performance progresses in unison, suggesting a potential link between the generalizations behind them. Children present remarkable consistency in their patterns of language acquisition. They often acquire linguistic phenomena in a similar order (Kuhl et al., 1992; Ingram, 1989), and make similar generalizations and over-generalizations (Kuczaj II, 1977; Pinker, 1995). This consistency provides an important starting point for linguistic study. For example, arguments in favor of single or dual system accounts of morphological representation are often backed by computational models of children learning trajectories (e.g., Rumelhart and McClelland, 1986; Pinker and Prince, 1988; Kirov and Cotterell, 2018). In this paper, we embrace this program for the study of computational language models, investigating learning trajectories. 1 The representations that language models (LM) acquire have been studied extensively, including studying their learning dynamics to improve training (see 6). However, very little work aimed at drawing connections between the training dynamics and the learned representations. In this work we adopt a behavioral approach, thus revealing that NLMs share learning trajectories and generalize in similar ways during training. This implies that studying trajectories of NLMs is worthwhile, in the sense that results on one architecture or size are expected to be reproducible by others. These findings call for a characterization of these trajectories, a new and promising territory for research. We take first steps to explore these directions, emphasizing their potential benefit to a better future understanding of what models learn. Specifically, we train NLMs on next-word prediction, but evaluate and compare them by tracking their performance on grammar learning in English, using the BLIMP dataset (See 2.1). BLIMP is a dataset that consists of 67K minimal pairs, where each pair includes a grammatically correct and a grammatically erroneous sentence. NLMs are tested for their ability to assign higher probability to the correct one. See example in Table 1, and details of our experimental methodology in 2. We begin (3) by establishing that NLMs learn grammatical phenomena in a consistent order. We evaluate NLMs at different time points along their training, showing that the performance on linguis-1 Code is supplied in https://github.com/borgr/ ordert 8281 Challenge Correct Erroneous Animate subject Galileo had talked to Bell. tic phenomena across initializations is highly correlated. We further find many similarities in the set of examples that they correctly classify. Still, models of different architectures learn at a different pace, and hence cannot be directly compared at identical time points. In 3.3, we overcome this by re-scaling the timeline. We then show that despite architectural differences, NLMs present highly correlated performance trajectories. In 3.4, we further demonstrate that even the choice of training data has minor influence on the results. Finally, in 3.5 we show that the learning dynamics essentially follows a single dimension. Namely, where the average performance is similar, success on linguistic phenomena is also similar. We proceed by analyzing the early stages of learning in 4. We find that, at first, NLMs rely mostly on local cues and not on word order. They thus resemble bag-of-words models over a window of the preceding tokens. Later stages seem to drift further away from bag-of-words models toward n -gram models, and with time seem to be more sensitive to structural cues. We also find evidence that some latent features that the model learns may not be related to linguistic phenomena. Finally, in 5 we take the first steps in categorizing linguistic phenomena by their learning trajectories. We identify links between their representations by finding phenomena that progress in unison. For example, we find that morphological phenomena are mostly learned at similar stages. Of particular interest are cases where performance decreases with time, which may suggest either overgeneralization or biases in the BLIMP challenges. We use BLIMP (Warstadt et al., 2019) to assess the extent to which generalizations are made by the NLMs. BLIMP includes 67 grammatical challenges categorized into 13 super-phenomena (e.g., island-related or quantifiers) comprising of 4 broad fields (e.g., Syntax, Semantics). Each challenge consists of 1K minimal pairs of sentences. A minimal pair contains a sentence and a near-duplicate distractor that incorporates an error on a particular linguistic phenomenon, i.e., only the phenomenon in question is changed between the sentences in a pair (see Table 1). Each challenge includes pairs with the same linguistic phenomenon. LM details: as training multiple GPT2 instances (Radford et al., 2019) is computationally demanding, we train smaller NLMs. Following Turc et al. (2019), we trained 1 instance of GPT2 small (width 768 , 12 layers, 8 attention heads) and 4 instances of GPT2 tiny (width 512 , 4 layers, 4 attention heads), with different random seeds. Similarly, we train a small TransformerXL (Dai et al., 2019), XL small (width 512 , 4 layers, 8 attention heads) and a full-sized one (width 4096 , 18 layers, 16 attention heads). We stop the full model after 600K steps, while the perplexity remained high. We use it for comparison to the early stages of learning of TransformerXL. All models' hyperparameters can be found in App. B. We also use the results of the fully trained GPT2, TransformerXL, LSTM and human performance reported in Warstadt et al. (2019). In 4, we compare NLMs with simpler models. To this end, we create two GPT2 tiny variations, denoted BOW and Window-5 . BOW replicates GPT2 tiny , but relies only on bag of words. This is achieved by removing the positional weights, and replacing the attention weights with a simple average. 2 Window-5 similarly ignores the positions, and additionally only attends to the last 5 words. Note that both are unidirectional LMs and consider only previously predicted words at each step. Unless explicitly stated otherwise (as in 3.4), all models were trained on the WikiBooks dataset (Zhu et al., 2015), which contains the English Wikipedia ( 2 . 1 B words) and BookCorpus ( 854 M words). This dataset resembles BERT's training data (Devlin et al., 2019), except that current Wikipedia is used. Additionally, we trained models on the following datasets: English openSubtitles (Lison and Tiedemann, 2016), newsCrawl (Barrault et al., 2019), GigaWord (Napoles et al., 2012), and 2 Supposedly, removing the positional embeddings would suffice. Empirically, it has little effect. Presumably, as embeddings only attend to previous positions, the network manages to represent positions by the difference between them. This is in line with the finding that GPT2's positional embeddings are not meaning-bearing (Wang and Chen, 2020). a sample of openWebText (3B words; Gokaslan and Cohen, 2019)  a replication of GPT2 dataset. Throughout this paper, we report Pearson correlation. Using Spearman correlation leads to qualitatively similar conclusions. When multiple models are correlated against each other, their average pairwise correlation is reported. In this section, we examine various aspects of NLMs, generally showing that their learning trajectories are similar. We evaluate network similarity by adopting a behavioral approach. Accordingly, networks are viewed as functions, whose latent features manifest themselves only by their influence on the network's behavior. Latent features are the unobserved causes of the measured behavior. Consequently, parameters, activation patterns and representations can be completely different among similar models. This is unlike the approaches employed by Williams et al. (2018); Saphra and Lopez (2019); Liu et al. (2021), which analyze internal representations directly. To formalize the above notion, let L t denote a checkpoint, the language model L at time t . Let pv ( L t ) denote its performance vector  the accuracy obtained by L on each BLIMP challenge p : pv ( L t ) = [ acc ( L t , p )] p  BLIMP  R 67 (1) Time t is measured in training steps or perplexity. The trajectory of the performance vector as a function of t reflects L 's training dynamics. Given this behavioral definition, we focus on comparing the relative strength of models. Similarity between models is thus measured as the correlation between their performance vectors. Hence, models are similar if they rank phenomena in the same way. On the other hand, models of the same average performance can be dissimilar: consider two models that agree on everything except nouns. One generates only feminine nouns and the other plural nouns. The models' average performance is similar, but due to their biases, they are correct on different challenges. This dissimilarity suggests that the models rely on different latent features. We begin by showing that models produced by different initializations learn the same phenomena, in the same order. In terms of our definitions above, this may imply that despite converging to different parameter values, the learned latent features and the generalization patterns made are similar. In order to examine the hypothesis empirically, we compute the correlation between 4 random initializations (Fig. 1). Results confirm the hypothesis, the correlation between GPT2 tiny instances is extremely high. It is already high after 10K steps, and remains high throughout training. We note that the correlation at step 0 is 0 (not shown), and that after 10K warm-up steps the network's ability as a LM is still poor. For example, perplexity is 10.9 after 10K steps and 6.7 after 70K steps. Next, we show that different architectures also present similar trajectories. As the learning pace is not comparable across models, computing correla-8283 tion in fixed and identical intervals is not informative. Instead, we choose t to be the perplexity on the development set, comparing models at the same performance level. TransformerXL is not directly comparable as perplexity requires the vocabulary to be the same. Following this paradigm, we see that GPT2 small and GPT2 tiny are highly correlated (> 0 . 9 ), presenting similar learning order throughout training. Observing the trajectories per challenge qualitatively, we see that they align very well (cf. Fig. 2 and App. A, C). TransformerXL also seems to share the general tendencies of the GPT2 architectures. Interestingly, we see that models behave similarly not only in terms of relative performance, but also at the example level (binary decision per minimal pair). We find that GPT2 small and GPT2 tiny have an average agreement of  = 0 . 83 (Fleiss et al., 1969). This implies strong consistency in the order of learning of different examples also within phenomena. Henceforth, we focus on the phenomena-level as it is more interpretable, lending itself more easily to characterization. We discuss per-example similarity further in App. D. So far, we have observed the common trajectories presented by NLMs that are trained in parallel. We proceed to compare trajectories of one model to other models' performance vectors at a single point of interest in their learning, i.e. a checkpoint's performance vector. This allows us to analyze how similarities evolve, rather than whether two trajectories are synced. We compare fully trained off-the-shelf NLMs with the trajectory of GPT2 tiny (Fig. 3a) and GPT2 small (App. E). The observed similarity to off-the-shelf models is high (0.6-0.8), implying that NLMs in general share tendencies and biases. Moreover, similarity increases until the point of same performance and then (when relevant) decreases. This suggests that the small NLM approaches off-the-shelf tendencies as it improves and stops somewhere on the same trajectory of generalizations (cf. 3.5). Furthermore, we find considerable correlation with the performance levels of humans on the different challenges, but still, all NLMs correlate better with our model than humans correlate with it. These results present a curious order imposed on the NLMs. Both GPT2 tiny and GPT2 small (App. E) are more similar to the LSTM model than to TransformerXL, and even less similar to GPT2 large . Interestingly, our models are more similar to an RNN and a model with a different architecture, than to a larger model with the same architecture. Thus, it seems that the architecture type cannot explain the similarities in the relative order. We further examine this issue in the next section. This section examines the possibility that the similarities reported in Fig. 3a can simply be explained by the similarity in the NLM's training data. More specifically, since the ranking by model similarity reported above fits the similarity between the training sets that the models were trained on, we view it as a potential confound and attempt to control for it. Our training data (WikiBooks) consists mostly of Wikipedia and so do the LSTM's and TransformerXL's training sets, which are trained on earlier versions of Wikipedia and WikiMatrix (Schwenk et al., 2019) respectively. GPT2, on the other hand, is trained on openWebText, which consists of scraped web pages. To tease apart the effect of training data, we trained 3 additional GPT2 tiny instances over the openWebText, openSubtitles and newsCrawl datasets. Results (Fig. 1) show that the dataset has more effect on the correlation than initialization. Hence, the choice of training data does affect the learning trajectory, but its effect decreases with training (correlation gets higher with more training steps). We also recompute the correlations from 3.3 after training GPT2 tiny on the same data as GPT2 large (App. F), and find that the relative order between the NLMs remains the same, with GPT2 large being the least similar. We conclude that while the training data affects the learned generalizations, it only very partially explains the observed similarities between NLMs. Based on the findings of the previous sub-sections, we hypothesize that current NLMs all learn in a similar order, where the effect of training data and architecture is secondary. In other words, training time, size and efficiency may affect what a model has learned, but not its learning order. This implies that stronger models may improve performance, but still follow a similar learning trajectory. If this hypothesis is correct, models should be most similar to models with the same performance; similarity should drop as the gap in performance widens. Controlled comparison supports this hypothesis. Fig. 3b presents the correlation of GPT2 tiny training trajectory with several static checkpoints taken during GPT2 small training. We observe that at the point in which the average performance of GPT2 tiny is closest to that of the checkpoint, the correlation peaks, and then decreases again as GPT2 tiny surpasses the checkpoint in average performance. So overall correlation peaks when average performance is most similar. Note that despite the different network sizes and convergence rates, the correlation's maximal value is very high (higher than 0.9). Further experiments show similar trends. Fig. 3a presents a similar investigation, albeit with more varied architectures and training datasets. Here too the maximum correlation is obtained around the point of most similar performance. NLMs are most similar to other NLMs with the same performance. However, when compared to non-neural LMs, this is no longer the case. More specifically, we compare GPT2 tiny to two 5-gram LMs trained on the same dataset as the NLMs (WikiBooks) and another (GigaWord) dataset. Results are shown in Fig. 4, which is qualitatively different from Fig. 3a. Here, similarity in performance implies neither high correlation, nor the point of highest similarity. This serves both as a sanity check to our methodology, and as a reminder of model biases: In general, models may have different biases and tendencies, regardless of overall performance. In our case, it seems that NLMs share biases between them that are not necessarily shared with other LMs. While not the main purpose of the analysis, our comparison reveals other noteworthy trends. For example, 5-gram LMs trained on different corpora have different correlations to the GPT2 tiny trajectory. This is further discussed in App. G. We find that the order of learning is surprisingly stable across architectures, model sizes and training sets. Therefore, given a new NLM, the order 8285 in which it will learn linguistic phenomena can be predicted by another model that achieves a similar average accuracy. When considering non-neural LMs, this observation does not always hold: inherently different architectures (such as 5-grams) have very different trajectories. Hence, future models with very different induced biases may present different orders. Having established that different NLMs learn in a consistent order, we investigate the emerging learning trajectory by comparing it with simpler reference models. Our goal is to identify distinct learning phases that characterize NLM's training. Setup. We compare GPT2 tiny to fully trained LMs (same as 3.3), as well as to a variety of metrics. For each metric m we compute the average score over each example for each of the 67 sets E p i  p [ m ( p i )]  R 67 . The results are replicated with GPT2 small and TransformerXL and lead to similar conclusions (see App. E). Sentence-level Metrics. First, we consider two sentence-level metrics: sentence length (in tokens) and syntactic depth. Assuming a sentence parse tree, the depth is the longest path from a word to the root. Sentence length is often considered to be a source of challenge for infants (Brown, 1973) and networks (Neishi and Yoshinaga, 2019), regardless of the sentence's complexity. Syntactic depth (Yngve, 1960) is a measure used to assess how cognitively complex a sentence is. We leave the question of which measure of linguistic complexity (Szmrecsnyi, 2004) correlates best with the trajectory exhibited by NLMs to future work. Our results (Fig. 5) show that neither sentence-level metric (length and syntactic depth) can predict well what is difficult for the model. This is not surprising, as both measures only capture sentence complexity at a general level, and are not directly related to the linguistic phenomenon that is being tested. We do see that the syntactic depth starts off as a worse predictor of the NLM performance and ends as a better one. We provide a different perspective on this initial learning phase, before and after that switch, later in this section. Next, we compare the performance vector with task difficulty for humans, as reported in the original BLIMP paper. We observe that correlation is fairly high after a sufficient number of steps. In fact, the network becomes more similar to humans as it improves: at the beginning, the network relies on different features than humans, but with time more of the hurdles are shared. However, correlation saturates at a mid-range correlation of under 0.5. This suggests that the network (partially) relies on features that are not used by human annotators. These may be valid generalizations not tested by BLIMP, or erroneous ones that are still beneficial to reduce the score on the task it was trained on (cf. McCoy et al., 2019). We revisit this issue in 5. Comparison with Limited Context and Locality. Our methodology opens the door to examine other potential biases of LMs. We now do so, starting with context and locality. We consider models that take into account different scopes of context: unigram, and 2-5 gram LMs that can exploit the order of preceding words. We argue that the correlation between NLMs and n -gram LMs may indicate that features based on limited context are also employed by NLMs. Surprisingly, the unigram model, which doesn't use context, perfectly classifies 7 phenomena, achieves 98.1% accuracy on 1, and completely fails (0% accuracy) on 8. This suggests that high accuracy on some syntactic and semantic challenges (as defined by BLIMP) can be achieved by simple heuristics. Note, however, that the NLMs we test are not trained towards any specific phenomena and are not fine-tuned in any way. Hence, NLMs can only attain heuristics or biases (generalization errors) which are beneficial in general, not ones specific to our test challenges. While NLMs initially present a strong correlation with the unigram model, this correlation quickly drops (see Fig. 5). From the outset, GPT2 tiny succeeds on 6 of the 8 phenomena that are classified well by unigrams, and 4 of the 8 that 8286 the unigram model utterly fails on. Interestingly, for 3 of the other phenomena on which the unigram failed, GPT2 tiny initially achieves 0% accuracy (chance level is 50%), but its accuracy does climb during training (e.g., see App. A). We conclude that, as expected, the NLM acquires a bias towards predicting frequent words early in training, but that this bias is weighed in against other (contextual) considerations later on in training. Comparing different scopes of context, our results (Fig. 6 and App. E) show that throughout training, the network presents high correlation with n -gram models. From a certain point onward, the network becomes more similar to the bi-gram model than to the other n -gram LMs. We also note that similarity peaks early on, but with time the correlation decreases. This may suggest that initially, the NLMs acquire grammatical behavior that resembles a Markov model, or even a bi-gram model. Only later does the network rely more on global features. This is in line with our earlier findings, which show an increasing correlation with syntactic depth as compared to sentence length. At the very beginning, NLMs often generate one word repetitions (e.g., "the" Fu et al., 2020). This seems to be at odds with our finding that grammar learning already begins at this early stage. However, while frequency may dictate the most probable predictions, comparing two options that differ only slightly may prove to depend more on context, as our results indicate. comparing NLMs to n -grams, we examined the effect of context within a fixed window size. Now we examine the effect of word order, within a window and in general. To this end, we create two ablated GPT2 tiny models. BOW is agnostic to the order between preceding tokens, while Window-5 is similar but relies only on 5 tokens (details in 2). Our results suggest that initially, the identity of the preceding words is more important than their order. Both BOW and Window-5 better correlate with our NLM than the n -gram models. Later on, this trend reverses and the n -grams, that do exploit word order, become better correlated. Furthermore, the correlation with Window-5 is sig-nificantly smaller than with BOW at later stages of learning, suggesting that the network gradually learns to rely on more context (cf. Saphra and Lopez, 2019). To understand the latent features learned by NLMs, we categorize linguistic phenomena through the lens of their learning trajectories. We ask whether linguistically similar phenomena are learned in a similar fashion, and whether what is learned similarly is defined by linguistic terms. We inspect linguistic categories by comparing the learning trajectories of their phenomena. In the Morphology field, we find that they display similar gradual curves, ultimately reaching high performance (median accuracy 0 . 85 , see Fig. 7a). This may indicate that some latent features learned are morphological, and affect performance on almost all 'Morphology' phenomena. Syntax-semantics phenomena also present unique behavior: their scores plateau near chance performance (see Fig 7b), suggesting that the learned features are insufficient to correctly represent phenomena in this field. The other fields, "se-mantics" and "syntax" (Figs 7c,7d), do not present prototypical learning curves, suggesting that they are too broad to correspond to a single learning pattern. This, in turn, may suggest that they do not all correspond to a well-defined set of latent features. Next, we follow the reverse direction and cluster the learning curves of GPT2 tiny . We use spectral clustering with 10 clusters and sklearn default parameters, by projecting the learning curves into a normalized Laplacian and applying k-means. Intuitively, learning curves with similar values along the principal directions, are clustered together. Other clustering methods show similar results. learning profiles, some more expected than others. For some, accuracy improves as learning progresses (see Fig. 8a). Some are barely learned, and accuracy remains at near-chance level (see Fig. 8b). Perhaps more surprisingly, some clusters deteriorate, and accuracy drops to nearly 0 as learning progresses (see Fig. 8c). Notably, some challenges are quite easy  NLMs instantly reach perfect accuracy (see Fig. 8d), while some are confusing  NLMs performance is worse than chance (see Fig. 8c). In the latter cases, the NLMs presumably learn unrelated, harmful generalizations. When inspecting the emerging clusters, many (but not all, see Fig. 8b) contain a shared prominent field, but often varied super-phenomena (see Fig. 8a). Thus, while the categorization in BLIMP reflects a common linguistic organization of grammatical phenomena, from the perspective of learning trajectories only few of the super-phenomena in BLIMP show consistent behavior. We cautiously conclude that there is some discrepancy between the common linguistic categorization of grammatical phenomena and the categorization induced by the learning trajectories of NLMs. An interesting direction for future work would therefore be the development of a theory that can account for the patterns presented by NLMs' learning trajectories. by a simple rule, easily learnable by an n -gram model. For example, in "principle A case 1", always preferring subjective pronouns (e.g., "she" or "he") over reflexive ones (e.g., "himself", "herself") is sufficient to obtain a perfect score, and preferring "not ever" over "probably/fortunately ever" solves "sentential negation NPI licensor present". The fact that NLM performance deteriorates, fits our finding that nascent NLMs resemble an n -gram model. Characterizing what networks learn is a longstanding challenge. Recently, studies suggested methods to analyze trained models such as probing (Tenney et al., 2019; Slobodkin et al., 2021), analyzing attention heads (Voita et al., 2019; Abnar and Zuidema, 2020) and neurons (finding also correlations across epochs; Bau et al., 2018) and assessing the extent to which LMs represent syntax (van Schijndel et al., 2019). Other works compare outputs, like us, to assess network generalizations (Choshen and Abend, 2019; Ontan'on et al., 2021), look for systematic biases (Choshen and Abend, 2018; Stanovsky et al., 2019) or evaluate characteristics of outputs (Gehrmann et al., 2021; Choshen et al., 2020). McCoy et al. (2020) fine-tuned BERT and tested generalizations on the adversarial dataset HANS (McCoy et al., 2019), finding models to make inconsistent generalizations. Their results differ from ours, but so is their setup, which in-8288 volves fine-tuning for inference. Characterizing the features learned by networks according to the order in which examples and phenomena are learned is a relatively new topic. Recently, Hacohen et al. (2020); Hacohen and Weinshall (2021); Pliushch et al. (2021) showed that classifiers learn to label examples in the same order. While their focus was on computer vision, it provided motivation for this work. Other studies use learning dynamics as a tool, rather than a topic of study. They choose training examples (Toneva et al., 2018), categorize examples (Swayamdipta et al., 2020) or characterize the loss-space (Xing et al., 2018). Little research on NLM learning dynamics and generalization types was previously conducted. Perhaps the closest to this work is Saphra and Lopez (2019), which compared LSTM representations with 3 types of linguistic tagger outputs, finding that correlation is low and that later in training, more context is used. The latter is reminiscent of our findings in 4. In parallel work, Liu et al. (2021) probe models during training. They show that, early in training, information required for linguistic classifi-cations is found somewhere in the layers of the model. Our work supports their findings by showing that grammar learning experiments conducted with one model are likely to replicate on another. Our methodology differs from theirs in requiring the information the model learnt to manifest itself in behavior rather than to be extractable with a dedicated classifier. Studying the trajectories of language learning is a mostly untapped area in NLP, but is a long-established field of research in linguistics and psychology. Such lines of research study topics such as acquisition of phonemes (Kuhl et al., 1992), morphology (Marcus et al., 1992), complex constructions (Gropen et al., 1991; Qing-mei, 2007) and innate learning abilities (Tomasello, 2003). Considerable computational work was also done on constructing models that present similar learning trajectories to those of infants (McClelland and Rumelhart, 1981; Perfors et al., 2010; Abend et al., 2017, among many others). Our work suggests that the generalizations NLMs make are coupled with the bottom-line performance. This gives a new angle and opens avenues of research when combined with previous work about bottom-line performance. For example, the bottom-line performance of small models could predict the performance of larger models (Ivgi et al., 2022). In such cases, the type of generalizations made might also be predicted from the smaller models. Our work is also closely related to fields such as curriculum learning (Bengio et al., 2009; Hacohen and Weinshall, 2019), self-paced learning (Kumar et al., 2010; Tullis and Benjamin, 2011), hard data mining (Fu and Menzies, 2017), and active learning (Krogh and Vedelsby, 1994; Hacohen et al., 2022; Ein-Dor et al., 2020). In these fields, the order in which data should be presented to the learner is investigated. On the other hand, in our work, we study the order of the data in which the learner is learning  which may shed some light on the advancement of such fields. We showed that NLMs learn English grammatical phenomena in a consistent order, and subsequently investigated the emerging trajectory. Our findings suggest that NLMs present consistent and informative trends. This finding suggests a path for studying NLMs' acquired behavior through their learning dynamics, as a useful complementary perspective to the study of final representations. Future work will consider the impact of additional factors, architectures and learning phases that appear only later in training. We hope that this work will increase the affinity between the knowledge and methodologies employed in developmental studies, and those used for studying NLMs. Our goal is to obtain a better understanding of what makes linguistic generalization complex or simple to learn, for both humans and NLMs. We thank Prof. Inbal Arnon for her helpful discussions. This work was supported in part by the Israel Science Foundation (grant no. 2424/21), by a grant from the Israeli Ministry of Science and Technology, and by the Gatsby Charitable Foundations.
abstain abstain result abstain method method abstain result abstain abstain abstain abstain abstain objective abstain abstain result abstain abstain objective result abstain abstain result method other abstain result abstain abstain result objective result abstain method objective abstain abstain result objective result result abstain method abstain abstain abstain abstain method method method method method abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method other other other other objective other other other abstain abstain other other abstain objective other other other abstain other other other other other method objective method abstain abstain method objective other other