Unsupervised learning of natural languages Zach Solan * , David Horn * , Eytan Ruppin , and Shimon Edelman * School of Physics and Astronomy and School of Computer Science , Tel Aviv University , Tel Aviv 69978 , Israel ; and Department of Psychology , Cornell University , Ithaca , NY 14853 Edited by James L. McClelland , Carnegie Mellon University , Pittsburgh , PA , and approved June 14 , 2005 -LRB- received for review December 25 , 2004 -RRB- We address the problem , fundamental to linguistics , bioinformatics , and certain other disciplines , of using corpora of raw symbolic sequential data to infer underlying rules that govern their production . Given a corpus of strings -LRB- such as text , transcribed speech , chromosome or protein sequence data , sheet music , etc. -RRB- , our unsupervised algorithm recursively distills from it hierarchically structured patterns . The ADIOS -LRB- automatic distillation of structure -RRB- algorithm relies on a statistical method for pattern extraction and on structured generalization , two processes that have been implicated in language acquisition . It has been evaluated on artificial context-free grammars with thousands of rules , on natural languages as diverse as English and Chinese , and on protein data correlating sequence with function . This unsupervised algorithm is capable of learning complex syntax , generating grammatical novel sentences , and proving useful in other fields that call for structure discovery from raw data , such as bioinformatics . computational linguistics grammar induction machine learning protein classification language acquisition any types of sequential symbolic data possess structure that is -LRB- i -RRB- hierarchical and -LRB- ii -RRB- context-sensitive . Natural-language text and transcribed speech are prime examples of such data : a corpus of language consists of sentences defined over a finite lexicon of symbols such as words . Linguists traditionally analyze the sentences into recursively structured phrasal constituents -LRB- 1 -RRB- ; at the same time , a distributional analysis of partially aligned sentential contexts -LRB- 2 -RRB- reveals in the lexicon clusters that are said to correspond to various syntactic categories -LRB- such as nouns or verbs -RRB- . Such structure , however , is not limited to the natural languages ; recurring motifs are found , on a level of description that is common to all life on earth , in the base sequences of DNA that constitute the genome . We introduce an unsupervised algorithm that discovers hierarchical structure in any sequence data , on the basis of the minimal assumption that the corpus at hand contains partially overlapping strings at multiple levels of organization . In the linguistic domain , our algorithm has been successfully tested both on artificialgrammar output and on natural-language corpora such as ATIS -LRB- 3 -RRB- , CHILDES -LRB- 4 -RRB- , and the Bible -LRB- 5 -RRB- . In bioinformatics , the algorithm has been shown to extract from protein sequences syntactic structures that are highly correlated with the functional properties of these proteins . The ADIOS Algorithm for Grammar-Like Rule Induction In a machine learning paradigm for grammar induction , a teacher produces a sequence of strings generated by a grammar G0 , and a learner uses the resulting corpus to construct a grammar G , aiming to approximate G0 in some sense -LRB- 6 -RRB- . Recent evidence suggests that natural language acquisition involves both statistical computation -LRB- e.g. , in speech segmentation -RRB- and rule-like algebraic processes -LRB- e.g. , in structured generalization -RRB- -LRB- 7 11 -RRB- . Modern computational approaches to grammar induction integrate statistical and rulebased methods -LRB- 12 , 13 -RRB- . Statistical information that can be learned along with the rules may be Markov -LRB- 14 -RRB- or variable-order Markov -LRB- 15 -RRB- structure for finite state -LRB- 16 -RRB- grammars , in which case the EM algorithm can be used to maximize the likelihood of the observed data . Likewise , stochastic annotation for context-free grammars -LRB- CFGs -RRB- can be learned by using methods such as the Inside-Outside algorithm -LRB- 14 , 17 -RRB- . www.pnas.org cgi doi 10.1073 pnas .0409746102 M We have developed a method that , like some of those just mentioned , combines statistics and rules : our algorithm , ADIOS -LRB- for automatic distillation of structure -RRB- uses statistical information present in raw sequential data to identify significant segments and to distill rule-like regularities that support structured generalization . Unlike any of the previous approaches , however , ADIOS brings together several crucial conceptual components ; the structures it learns are -LRB- i -RRB- variable-order , -LRB- ii -RRB- hierarchically composed , -LRB- iii -RRB- context dependent , -LRB- iv -RRB- supported by a previously undescribed statistical significance criterion , and -LRB- v -RRB- dictated solely by the corpus at hand . Consider a corpus of sentences -LRB- more generally , sequences -RRB- over a lexicon of size N , whose units in the case of language are words -LRB- starting with phonemes or letters , or even letters in a condensed text without spaces also works -RRB- . The algorithm starts by loading the corpus onto a directed pseudograph -LRB- a nonsimple graph in which both loops and multiple edges are permitted -RRB- whose vertices are all lexicon entries , augmented by two special symbols , begin and end . Each corpus sentence defines a separate path over the graph , starting at begin and ending at end , and is indexed by the order of its appearance in the corpus . Loading is followed by an iterative search for significant patterns , which are added to the lexicon as new units . The structure of the data graph and the operations carried out on it are illustrated in Fig. 1 . -LRB- See also Supporting Text , Figs. 4 17 , and Tables 1 12 , which are published as supporting information on the PNAS web site . -RRB- The algorithm generates candidate patterns by traversing in each iteration a different search path -LRB- initially coinciding with one of the original corpus sentences -RRB- , seeking subpaths that are shared by a significant number of partially aligned -LRB- 2 , 18 -RRB- paths . The significant patterns -LRB- P -RRB- are selected according to a context-sensitive probabilistic criterion defined in terms of local flow quantities in the graph , stated in BOX 1 : The Motif Extraction -LRB- MEX -RRB- Procedure . Generalizing the search path , the algorithm looks for an optional equivalence class -LRB- E -RRB- of units that are interchangeable in the given context -LSB- i.e. , are in complementary distribution -LRB- 2 -RRB- -RSB- . At the end of each iteration , the most significant pattern is added to the lexicon as a new unit , the subpaths it subsumes are merged into a new vertex , and the graph is rewired accordingly -LRB- two rewiring modes are available : a context-free Mode A and a context-sensitive Mode B , as described in BOX 2 : The ADIOS Algorithm -RRB- . The search for patterns and equivalence classes and their incorporation into the graph are repeated until no new significant patterns are found . The entire process is governed by the following three parameters : and , which control the definition of pattern significance , and L , which sets the width of the context window where equivalence classes are sought . We estimate the average-case computational complexity of the ADIOS algorithm empirically to increase linearly with the size of the corpus -LRB- see section 7 of Supporting Text and Fig. 17 -RRB- . The final lexicon includes those of the original symbols not incorporated into larger units and a forest of tree-structured root patterns distilled by the algorithm -LRB- that is , the patterns that reside This paper was submitted directly -LRB- Track II -RRB- to the PNAS office . Abbreviations : ADIOS , automatic distillation of structure ; CFG , context-free grammar ; MEX , motif extraction algorithm . To whom correspondence should be addressed . E-mail : se37@cornell.edu . 2005 by The National Academy of Sciences of the USA PNAS August 16 , 2005 vol . 102 no. 33 11629 11634 COMPUTER SCIENCES Fig. 1 . The graph structures used by the MEX and ADIOS algorithms . -LRB- A -RRB- The search path no. 1 -LRB- begin 3 e1 3 ... 3 e5 3 end -RRB- is rendered as a solid black line connecting the special begin and end vertices . Four other paths -LRB- nos. 4 , 5 , 6 , and 7 -RRB- join it along the vertices e2 , e3 , e4 , thus forming a bundle that may constitute a significant pattern subject to the MEX criterion described in BOX 1 : The MEX Procedure . Values of PR and PL , originating at e1 and e4 , respectively , are displayed for the example shown here . -LRB- B -RRB- A significant pattern -LRB- P e2 3 e3 3 e4 -RRB- has been identified . -LRB- C -RRB- A new vertex is added to the graph , replacing the elements subsumed by P. Paths that belong to sequences not subsumed by it , such as no. 3 here , are left untouched . -LRB- D -RRB- The path is generalized : the algorithm picks among the set of path segments encountered in a window of size L 4 those that differ in a single slot and are embedded in a common context -LRB- the relevant vertices are marked by open circles -RRB- . The vertices in this slot form an equivalence class E to be treated as a single unit , resulting in the first generalization step -LRB- see BOX 2 : The ADIOS Algorithm -RRB- . -LRB- E -RRB- The just-detected E -LRB- i 2 -RRB- is used to find an additional equivalence class ; it is specific to the current common context , thus enhancing the safety of generalization . -LRB- F -RRB- Stacking two equivalence classes leads to further generalization -LRB- see BOX 2 : The ADIOS Algorithm for details -RRB- . on the final graph , at the top level of the hierarchy -RRB- . Each pattern is structured as a tree because of the hierarchical process of pattern creation ; the leaves -LRB- terminals -RRB- are the original members of the lexicon , and the intermediate nodes are other patterns and equivalence classes -LRB- Fig. 2 -RRB- . The tree structure excludes cyclic recursion -LRB- loops -RRB- of patterns , although recursion may be introduced through pattern matching in a postprocessing stage . The final graph includes as many paths as all of the original sentences , but it can also generate many new ones . To generate a sentence from a chosen path in the graph , all its root patterns are traversed . Each recursively encountered pattern is treated as a derivation or parse tree -LRB- 19 -RRB- : it is read from top -LRB- root -RRB- to bottom -LRB- terminals -RRB- and from left to right , while accruing the terminals -LRB- words from the original lexicon -RRB- and selecting one member from each encountered equivalence class -LRB- Fig. 2C -RRB- . Because the equivalence relations only hold in the contexts specified by their parent patterns , the ADIOS representation is inherently safer than grammars that posit globally valid categories -LRB- such as `` parts of speech '' in a natural language -RRB- . At the same time , because each rewiring of the graph brings closer far-apart units that used to straddle the newly abstracted pattern , the resulting representation can capture long-range structural dependencies among units . Because patterns can be represented in the form of rewriting rules , which are context-free when Mode A is used -LRB- Fig. 2D -RRB- and context-sensitive when Mode B is used -LRB- Fig. 2G -RRB- , the end product of an ADIOS run constitutes a grammar . Because infinite recursion is not implemented in the current version of the algorithm , the representations learned by ADIOS are comparable in expressive power to finite grammars that are at least context free . This means that any grammar consisting of context-free rules can be loaded into an ADIOS instance -LRB- that is , translated into an ADIOS representation -RRB- , provided that a limit is placed on the number of times each rule is invoked recursively . In learning , the results described below show that our algorithm can acquire , from raw corpora , good operational approximations to those grammars that generate data rich with partially alignable sentences , including unconstrained natural-language data . Complex 11630 www.pnas.org cgi doi 10.1073 pnas .0409746102 grammars that are inherently ambiguous -LRB- 19 -RRB- because of the presence of multiple loops are dealt with effectively by acquiring more patterns . Results To date , the ADIOS algorithm has been tested on a variety of language data , as well as on DNA and protein sequences from several species . Details are available in Supporting Text . Language : Computational Grammar Induction . In the domain of language , we tested the ADIOS algorithm both on artificialgrammar data and on natural-language corpora such as ATIS -LRB- 3 -RRB- and CHILDES -LRB- 4 -RRB- and in languages as diverse as English and Chinese -LRB- 5 -RRB- . It is reasonable to require that the success of a learning algorithm be measured by the closeness , ideally , identity , of the learned and target grammars , G and G0 , respectively . Unfortunately , even for CFGs , equivalence is undecidable -LRB- 19 -RRB- . Moreover , for natural languages , G0 is inaccessible . We thus opt for testing our implementation for generativity as follows . In the artificial-grammar experiments , which start with a target grammar , a teacher instance of the model is first preloaded with this grammar -LRB- using the one-to-one translation of CFG rules into ADIOS patterns -RRB- , then used to generate the training corpus Ctraining . After training , the learner generates a test corpus Clearner and the teacher generates a test corpus Ctarget , the latter containing only novel sentences that do not appear in Ctraining . In testing a learned grammar G for strong generativity , the structural descriptions -LRB- parse trees -RRB- it assigns to novel strings are compared with those produced by the target grammar G0 ; an example of a state-of-the-art constituent learning algorithm that adopts this criterion can be found in ref . 20 . A weak generativity criterion requires merely that G accept novel G0-grammatical strings as such and reject the ungrammatical ones . Strong generativity of grammars acquired by unsupervised algorithms that work from raw data is in principle difficult to test , because of the unavailability of reliable `` gold standard '' structural descriptions for such data . At the same time , demonstrating even weak perfect generativity by an automatically acquired representation has until now proved elusive . Our results constitute significant progress on both fronts : the representations acquired by the ADIOS algorithm are -LRB- i -RRB- structured in a manner that makes certain syntactic sense , and -LRB- ii -RRB- generative in that they largely encode and produce acceptable sentences . Solan et al. . The two corpora , Clearner and Ctarget , then are used to calculate precision -LRB- the proportion of Clearner accepted by the teacher -RRB- and recall -LRB- the proportion of Ctarget accepted by the learner -RRB- . A sentence is accepted if it precisely fits one of the paths in the ADIOS graph -LRB- that is , it can be generated by the path -RRB- . In the natural language experiments , where no target grammar is available , the given corpus is split into two portions , one for training -LRB- Ctraining -RRB- and one for testing -LRB- Ctarget -RRB- , and the same evaluation method is applied , except that precision must in this case be evaluated by an external referee -LRB- e.g. , by a human subject -RRB- . This evaluation method is unique in that -LRB- i -RRB- it defines precision and recall more conservatively than is standard in the literature -LRB- 21 -RRB- , and -LRB- ii -RRB- it involves testing both the capability of the learner to accept all of the grammatical sentences and its capability to generate only sentences that the teacher would deem grammatical . We have conducted a series of experiments designed to evaluate the performance of ADIOS in grammar induction -LRB- Fig. 3 -RRB- . Learning a simple CFG . We first replicated an experiment -LRB- 22 -RRB- that aimed to reconstruct a specific CFG -LRB- 29 terminals and 7 rules -RRB- from a corpus of 2,000 sentences . Whereas the algorithm proposed by ref . 22 generated between 3,000 and 4,000 rules , ADIOS -LRB- used in the default Mode A -RRB- yielded 28 patterns and 9 equivalence classes and achieved 100 % precision and 99 % recall -LRB- see Solan et al. . section 2.1 of Supporting Text and Table 1 -RRB- . Next , we applied to a somewhat more complex CFG -LRB- TA1 grammar , 50 terminals and 28 rules -RRB- and showed that it performs well even when only 200 sentences are used for training , as shown in Fig. 3A -LRB- see section 2.2 of Supporting Text , Tables 2 5 , and Fig. 8 -RRB- . Learning a complex CFG . Because the ADIOS algorithm is greedy -LRB- the best available pattern in each iteration is immediately and irreversibly rewired -RRB- , the syntax it acquires depends on the order of sentences in the training set . This characteristic is expected to affect the learning of a complex CFG , especially if it contains many loops . To assess this dependence and to mitigate it , we train multiple learners on different order-permuted versions of the corpus generated by the teacher . As Fig. 3B illustrates , for the parameter values explored -LRB- L -LCB- 3,4,5,6 -RCB- ; 30 or 150 learners ; corpus size between 10,000 and 120,000 sentences -RRB- , the optimal precision-recall tradeoff for learning the ATIS CFG -LRB- 357 terminals and 4,592 rules -RRB- -LSB- B. Moore and J. Carroll -LRB- 2001 -RRB- , www.informatics.susx.ac.uk research nlp carroll cfg-resources -RSB- is obtained with a 150-learner cohort and L between 5 and 6 -LRB- see section 3.1 of Supporting Text , Table 7 , and Fig. 9 -RRB- . Generativity of the learned natural language grammar . To test the ability of ADIOS to generate acceptable novel sentences after learning from a natural language corpus , we trained it on 12,700 sentences from the ATIS-2 natural language corpus of size ADIOS PNAS August 16 , 2005 vol . 102 no. 33 11631 COMPUTER SCIENCES Fig. 2 . Progressive abstraction of patterns constructs a forest of trees rooted in vertices of the graph -LRB- training data generated by a CFG , TA1 ; see Supporting Text -RRB- . -LRB- A -RRB- Pattern P49 , consisting of the terminal the and equivalence class E50 -LCB- bird , cat , cow , dog , horse , rabbit -RCB- , is distilled . -LRB- B -RRB- Further application of the algorithm yields equivalence classes -LRB- underlined -RRB- such as E64 , which contain some verbs . -LRB- C -RRB- Pattern P116 can generate 896 novel sentences , eight of which appear in the training corpus -LRB- the generalization factor , 8 896 , appears in parentheses -RRB- . A novel sentence , such as that George is eager to read disturbs Joe , can be read off the leaf level of the tree -LRB- numbered arrows indicate traversal order during generation -RRB- . Pattern P116 is a root pattern , that is , a unit situated on a final path . -LRB- D -RRB- The set of context-free productions -LRB- rewriting rules -RRB- that is equivalent to the tree of pattern P116 . -LRB- E -RRB- The initial path through a sentence to which ADIOS was applied in the context-sensitive mode B. -LRB- F -RRB- The same path after three root patterns -LRB- P55 , P72 , and P178 -RRB- have been distilled . Note how the two similar but not identical root patterns , P55 and P72 , capture the difference between the equivalence classes E56 and E66 -LRB- indeed , Beth , for example , is equivalent to Jim in the context of P72 but not of P55 -RRB- . In this manner , ADIOS enforces long-range agreement between E56 and the phrase does n't she -LRB- embedded in P178 -RRB- and avoids overgeneralization . -LRB- G -RRB- The two context-sensitive rules in this example are -LSB- begin P55 3 begin E56 thinks that -RSB- and -LSB- P72 P178 3 E66 thinks that P178 -RSB- . Fig. 3 . Experimental results . -LRB- A -RRB- The performance of an ADIOS model trained on extremely small corpora generated by TA1 . Optimal combinations of recall and 0.6 , 0.01 , L 5 ; maximum recursion depth for the teacher here and below set to 10 -RRB- are shown for four different precision -LRB- single learner , 30 trials , conditions as follows : -LRB- i -RRB- the default learning mode A -LRB- context-free mode , see BOX 2 : The ADIOS Algorithm -RRB- ; with a 800-sentence training corpus -LRB- data not shown -RRB- , both precision and recall reach 90 % ; -LRB- ii -RRB- mode B -LRB- context-sensitive mode -RRB- ; -LRB- iii -RRB- a `` semantically supervised '' mode in which the equivalence classes of the target grammar are made available to the learner ahead of time -LSB- see evidence on the contribution of extra-linguistic knowledge to language acquisition -LRB- 42 -RRB- -RSB- ; -LRB- iv -RRB- bootstrap mode , starting from a letter level and training on corpora in which all spaces between words are omitted . To maintain a comparable level of performance , the bootstrap mode requires larger corpora -LRB- size , shown in parentheses : 200 10,000 sentences -RRB- . -LRB- B -RRB- Using the ATIS CFG -LRB- 4,592 rules -RRB- -LRB- 3 -RRB- as the teacher of multiple ADIOS learners . Precision is defined by the mean over learners , while for recall acceptance by one learner suffices . Several corpus sizes , context window widths L and numbers of learners are compared . -LRB- C -RRB- Output generated by an instance of ADIOS that had been trained on the natural-language ATIS-2 corpus was judged to be as acceptable to human subjects as sentences from ATIS-2 . Acceptability data -LRB- mean standard deviation -RRB- are from eight subjects . -LRB- D -RRB- A dendrogram illustrating the relationships among six different natural languages using pattern spectra . We define a pattern spectrum as the histogram of pattern types , whose bins are labeled by sequences such as -LRB- T , P -RRB- or -LRB- E , E , T -RRB- -LSB- E , equivalence class ; T , tree-terminal -LRB- original unit -RRB- ; P , significant pattern -RSB- . This plot was generated by applying hierarchical clustering to a table of Euclidean distances among histograms of patterns learned by ADIOS from online multilingual Bible texts -LRB- 5 -RRB- , each consisting of 31,100 verses -LRB- single learner per language -RRB- . 13,043 -LRB- 3 -RRB- and tested its recall level on the 343 remaining sentences . The small size of the training corpus results in a relatively low recall of 40 % -LRB- under our strict definition that requires an exact match -RRB- . Fig. 3C compares the acceptability of ADIOS-generated sentences with original sentences from the ATIS-2 corpus -LRB- see section 4.1 of Supporting Text and Fig. 10 -RRB- . Notably , the output generated by ADIOS is on the average as acceptable to human subjects as the original corpus sentences . The human-judged precision -LRB- 70 % , as shown in the plot -RRB- is remarkable ; for comparison , the ATIS-CFG grammar , handconstructed to fit the ATIS-2 corpus -LRB- with recall of 45 % on same data -RRB- produces 99 % ungrammatical sentences when used in a generative fashion . Structural language modeling . A language model such as a table of word n-gram probabilities or a probabilistically annotated grammar can be made to predict the next word in a sentence from its predecessors , an operation that is highly useful in a variety of natural language processing tasks -LRB- 23 -RRB- . A structured language model of the kind learned by ADIOS , which includes reliable syntactic information , can be especially effective , because it captures longer-range dependencies than unstructured -LRB- purely statistical -RRB- n-gram models . When used as a language model on the ATIS-2 corpus -LRB- 1,294-word lexicon ; 11,386 unique sentences -RRB- , ADIOS achieved perplexity of 11.5 -LRB- see section 4.2 of Supporting Text and Fig. 11 -RRB- . In comparison , automatically acquired regular grammars achieve perplexity between 30 and 40 ; 3-gram models that use sophisticated smoothing of probabilities result in perplexity of 14 -LRB- 24 27 -RRB- . Languages other than English . Applying ADIOS to the Parallel Bible -LRB- 5 -RRB- corpus , we compared six different languages through a metaanalysis of their respective ADIOS grammars -LRB- see section 4.3 Perplexity measures the degree of uncertainly about the next word , averaged over the test set -LRB- see ref . 23 -RRB- . The lower the perplexity , the better the model . of Supporting Text and Fig. 12 -RRB- . The dendrogram shown in Fig. 3D captures the resulting typological relationships . Bioinformatics . Although the present work focuses on language , ADIOS also has been tested on various problems in bioinformatics ; a representative result is reported here -LRB- another result is included in section 6.1 of Supporting Text , Tables 10 and 11 , and Fig. 15A -RRB- . We used the representations acquired by ADIOS from protein sequence data to define a space where each protein is represented by a vector of its root-patterns . We then proceeded to show that this space supports functional classification of proteins , focusing on a family of enzymes whose functionality is captured by a homology tree structure of four levels . Classification of the 6,751 enzymes belonging to this family was tested previously by the SVM-PROT system -LRB- 28 -RRB- , ** with the proteins described in terms of physical features -LRB- polarity , charge , etc. -RRB- of their building blocks . Despite using exclusively the raw sequence information , ADIOS attained classification performance comparable with that of the SVM-PROT system -LRB- success rate of 95 % -RRB- . The same performance was attained also at the third level of the hierarchy . This result implies that the root-patterns extracted from raw sequence data carry useful functional information . Discussion The ADIOS algorithm differs from other methods of grammar induction in the data it requires and the representations it builds , as well as in its algorithmic approach . Most existing methods require corpora tagged with part-of-speech information -LRB- 29 -RRB- ; the very few ** The function of an enzyme is encoded by the name given to it by the Enzyme Commission -LRB- EC -RRB- , which has the form n1 : n2 : n3 : n4 ; e.g. , 1:1:3:13 stands for alcohol oxidase . The present experiment concentrated on the oxidoreductases family -LRB- EC 1.x.x.x -RRB- . Protein sequences annotated with EC numbers were extracted from the SWISSPROT database -LRB- Release 40.0 , http : us.expasy.org sprot -RRB- ; sequences with double annotations were removed . All together , 6,751 proteins were analyzed . Classification was tested at level 2 -LRB- EC 1.x -RRB- and at level 3 -LRB- EC 1.x.x -RRB- . 11632 www.pnas.org cgi doi 10.1073 pnas .0409746102 Solan et al. . exceptions -LRB- 30 22 -RRB- are not known to scale up . The extraction of grammatical primitives in published methods may rely on collocation frequencies -LRB- 30 -RRB- or on global criteria such as the likelihood of the entire corpus given the grammar -LRB- 14 , 17 , 29 , 31 , 32 -RRB- . In comparison , ADIOS carries out its inferences locally , in the context provided by the current search path , alleviating the credit assignment problem in learning and making productive use of learned structures safer . In its reliance on transition probabilities , the ADIOS criterion for pattern significance may seem to resemble the well-known approaches that use various mutual information measures in defining syntactic constituents , such as the influential early work described in ref . 33 . Mutual information-based approaches , however , are limited by diverse issues such as choosing the context window and alleviating the data sparseness problem -LRB- 29 -RRB- , which our algorithm largely circumvents . In particular , when ADIOS is iterated , symbols that may have been initially very far apart are allowed to exert influence on each other , enabling it to capture long-range syntactic dependencies such as agreement -LRB- Fig. 2F -RRB- . Furthermore , ADIOS works with raw text or transcribed speech , successfully overcoming the data sparseness problem through the use of a statistically feasible local-context significance criterion . Although ADIOS makes no prior assumptions about the structures it seeks , the patterns and equivalence classes it learns can be translated in a straightforward manner into the form of rewriting rules . These representations are both expressive enough to support extensive generativity , and , in principle , restrictive enough to capture many of the structure-sensitive aspects of syntax -LRB- 1 -RRB- documented by linguists , such as tough movement -LRB- see Fig. 8 -RRB- . It is instructive to consider ADIOS in the context of the problem of language acquisition , which has long been a daunting challenge for cognitive scientists -LRB- 29 , 34 , 35 -RRB- . Because a completely bias-free unsupervised learning is impossible -LRB- 34 , 36 -RRB- , the real issue in language acquisition is to determine the model constraints . In our approach , the constraints are defined algorithmically in the form of a method for detecting units -LRB- patterns -RRB- that are hierarchically structured and supported by context-sensitive statistical evidence . When considered as a model of language acquisition , ADIOS is clearly incomplete , because it currently relies on syntactic regularities and leaves out conceptual knowledge and grounding of speech acts in the external events . Nevertheless , our approach is compatible with a range of findings concerning language acquisition , such as the use of statistical cues -LRB- 7 , 37 -RRB- and the importance of patternlike constructions -LRB- 38 40 -RRB- -LRB- many more links to linguistic theories are offered in Supporting Text -RRB- . Moreover , it performs well in a wide variety of situations that require unsupervised learning of structural information from untagged data . In grammar induction from large-scale raw corpora , our method achieves precision and recall performance unrivaled by any other unsupervised algorithm . It exhibits good performance in grammaticality judgment tests -LRB- including standard tests routinely taken by students of English as a second language -RRB- and replicates the behavior of human subjects in certain psycholinguistic tests of artificial language acquisition . Finally , the very same algorithmic approach also is proving effective in other settings where knowledge discovery from sequential data is called for , such as bioinformatics . BOX 1 : The Motif Extraction -LRB- MEX -RRB- Procedure Seeking a Pattern Along a Search Path . The MEX procedure is applied for each search path of the ADIOS graph that is to be explored for patterns . In the example of Fig. 1 A , this is path no. 1 : begin 3 e1 3 3 e5 3 end . Other paths may join and leave the search path at various vertices . In this example , the bundle of path sections between e2 and e4 display a certain coherence , possibly indicating the presence of a significant pattern . Two probability functions are defined over the graph for any given search path . The first one , PR , is the right-moving ratio of fan-through -LRB- through-going flux of paths -RRB- to fan-in -LRB- incoming Solan et al. . flux of paths -RRB- , which varies along the search path . Starting at e1 we define PR at e2 as PR -LRB- e1 ; e2 -RRB- -LRB- no . of paths from e1 to e2 -RRB- divided by -LRB- total no . of paths entering e1 -RRB- . At e3 it becomes PR -LRB- e1 ; e3 -RRB- -LRB- no . of paths from e1 through e2 to e3 -RRB- divided by -LRB- total no . of paths from e1 to e2 -RRB- . In Fig. 1 A , PR first increases because other paths join to form a coherent bundle , then decreases at e5 , because many paths leave the search path at e4 . To quantify this decline of PR , which we interpret as an indication of the end of the candidate pattern , we define a `` decrease ratio , '' DR , whose value at e4 is DR -LRB- e1 ; e4 -RRB- PR -LRB- e1 ; e5 -RRB- PR -LRB- e1 ; e4 -RRB- . We require that it be smaller than a preset `` cutoff parameter '' 1 . In a similar manner , we proceed leftward from some point down the search path -LRB- in the present example , starting at e4 -RRB- and examine the left-going ratio of fan-through to fan-in , PL . Thus , PL -LRB- e4 ; e3 -RRB- -LRB- no . of paths from e3 to e4 -RRB- divided by -LRB- total no . of paths entering e4 -RRB- and so on . The value of PL increases leftward ; the point e2 at which it first shows a decrease DL -LRB- e4 ; e2 -RRB- PL -LRB- e4 ; e1 -RRB- PL -LRB- e4 ; e2 -RRB- is interpreted as the starting point of the candidate pattern . The statistical significance of the decreases in PR and PL must be evaluated . PR and PL can be regarded as variable-order Markov probability functions . Continuing with the example of Fig. 1 , we define their significance in terms of a null hypothesis stating that PR -LRB- e1 ; e5 -RRB- PR -LRB- e1 ; e4 -RRB- and PL -LRB- e4 ; e1 -RRB- PL -LRB- e4 ; and e2 -RRB- , and require that the P values of both DR -LRB- e1 ; e4 -RRB- be , on average , smaller than a threshold 1 . DL -LRB- e4 ; e2 -RRB- A bundle of coinciding paths whose end-points obey these significance conditions is declared to be a candidate significant pattern . Given a search path , we calculate both PL and PR from all of the possible starting points -LRB- such as e1 and e4 in our example -RRB- , traversing each path leftward and rightward , correspondingly . This technique defines many search-sections , which may be candidates for significant patterns . The most significant one of all these candidates is returned as the outcome pattern for the search path in question . The Generalized Search Path . To generalize the notion of a search path , we consider a situation in which it contains an open slot where multiple alternative subpaths coexist within a fixed context defined by the main path . As an example , consider a window of size L 3 , composed of e2 , e3 , and e4 , with a slot at e3 . The generalized search path in this case consists of all of the paths that share the context e2 , e4 and branch into all possible vertices at location e3 . We thus P -LRB- e3 e2 ; e4 -RRB- , where each P -LRB- e3 e2 ; e4 -RRB- is define P -LRB- e3 e2 ; e4 -RRB- calculated by considering a different path going through the corresponding e3 . Likewise , we proceed to define P -LRB- e5 e2 e3 e4 -RRB- P -LRB- e5 e2 ; e3 ; e4 -RRB- and so on . In the example of Fig. 1D , the generalized path is defined over a window of length 4 with one slot at the third location ; the context at the other locations is fixed . In Fig. 1F , there are two slots , in locations 2 and 3 . BOX 2 : The ADIOS Algorithm 1 . initialization : load all sentences as paths onto a pseudograph whose vertices are the unique words of the corpus . 2 . pattern distillation : for each path -LRB- see Fig. 1 A C for an illustration -RRB- a. find the leading significant pattern perform MEX on the search path by considering all search segments -LRB- i , j -RRB- , j i , starting P R at e i and P L at e j ; choose out of all segments the leading significant pattern P for the search path . b. rewire graph CREATE a new vertex corresponding to P. Mode A -LRB- context free -RRB- : replace the string of vertices comprising P with the new vertex P on all paths on which it occurs -LRB- Fig. 1C -RRB- . PNAS August 16 , 2005 vol . 102 no. 33 11633 COMPUTER SCIENCES Mode B -LRB- context sensitive -RRB- : replace the string of vertices comprising P with the new vertex P only on those paths on which P is significant according to the MEX criterion . 3 . generalization first step for each path -LRB- Fig. 1 D and E -RRB- a. slide a context window of size L along the search path from its beginning vertex to its end ; at each step i -LRB- i 1 , K L 1 for a path of length K -RRB- examine the generalized search paths : for all j i 1 , , i L 2 do i. define a slot at location j ; ii . define the generalized path consisting of all paths that have identical prefix -LRB- at locations i to j 1 -RRB- and identical suffix -LRB- at locations j 1 to i L 1 -RRB- ; iii . perform MEX on the generalized path ; b. choose the leading P for all searches performed on each generalized path ; c. for the leading P define an equivalence class E consisting of all the vertices that appeared in the relevant slot at location j of the generalized path ; d. rewire graph CREATE a new vertex corresponding to P -LRB- Fig. 1E -RRB- and replace the string of vertices it subsumes with the new vertex P on all paths where it occurs . We list here , and in the next rewiring step , only mode A ; in mode B the replacement should occur only on the paths for which the new P is declared significant by MEX . 4 . generalization bootstrap for each path a. slide a context window of size L along the search path from its beginning vertex to its end ; at each step i -LRB- i 1 , K L 1 for a path of length K -RRB- do : 1 . Phillips , C. -LRB- 2003 -RRB- in Encyclopedia of Cognitive Science , ed . Nadel , L. -LRB- Macmillan , London -RRB- Vol . 4 , pp. 319 329 . 2 . Harris , Z. S. -LRB- 1954 -RRB- Word 10 , 140 162 . 3 . Hemphill , C. T. , Godfrey , J. J. & Doddington , G. R. -LRB- 1990 -RRB- in Proceedings of a Workshop on Speech and Natural Language , ed . Stern , R. M. -LRB- Morgan Kaufmann , San Francisco -RRB- , pp. 96 101 . 4 . MacWhinney , B. & Snow , C. -LRB- 1985 -RRB- J. Comput . Lingustics 12 , 271 296 . 5 . Resnik , P. , Olsen , M. B. & Diab , M. -LRB- 1999 -RRB- Computers Humanities 33 , 129 153 . 6 . Adriaans , P. & van Zaanen , M. -LRB- 2004 -RRB- Grammars 7 , 57 68 . 7 . Saffran , J. R. , Aslin , R. N. & Newport , E. L. -LRB- 1996 -RRB- Science 274 , 1926 1928 . 8 . Seidenberg , M. S. -LRB- 1997 -RRB- Science 275 , 1599 1603 . 9 . Marcus , G. F. , Vijayan , S. , Rao , S. B. & Vishton , P. M. -LRB- 1999 -RRB- Science 283 , 77 80 . 10 . Pena , M , Bonatti , L. L. , Nespor , M. & Mehler , J. -LRB- 2002 -RRB- Science 298 , 604 607 . ~ 11 . Seidenberg , M. S. , MacDonald , M. C. & Saffran , J. R. -LRB- 2002 -RRB- Science 298 , 553 554 . 12 . Pereira , F. -LRB- 2000 -RRB- Philos . Trans . R. Soc . London 358 , 1239 1253 . 13 . Geman , S. & Johnson , M. -LRB- 2003 -RRB- in Mathematical Foundations of Speech and Language Processing , IMA Volumes in Mathematics and Its Applications , eds . Johnson , M. , Khudanpur , S. , Ostendorf , M. & Rosenfeld , R. -LRB- Springer , New York -RRB- , Vol . 138 , pp. 1 26 . 14 . Stolcke , A. & Omohundro , S. -LRB- 1994 -RRB- in Grammatical Inference and Applications , eds . Carrasco , R. C. & Oncina , J. -LRB- Springer , New York -RRB- , pp. 106 118 . 15 . Guyon , I. & Pereira , F. -LRB- 1995 -RRB- in Proceedings of the Third International Conference on Document Analysis and Recogition , ed . Shen , C. Y. -LRB- IEEE Computer Society , Montreal -RRB- , pp. 454 457 . 16 . Gross , M. -LRB- 1997 -RRB- in Finite-State Language Processing , eds . Roche , E. & Schabes , ` Y. -LRB- MIT Press , Cambridge , MA -RRB- , pp. 329 354 . 17 . Lari , K. & Young , S. -LRB- 1990 -RRB- Computer Speech Language 4 , 35 56 . 18 . van Zaanen , M. -LRB- 2000 -RRB- in Proceedings of the 18th International Conference on Computational Linguistics , ed . Kay , M. -LRB- Saarbrucken , Germany -RRB- pp. 961 967 . 19 . Hopcroft , J. E. & Ullman , J. D. -LRB- 1979 -RRB- Introduction to Automata Theory , Languages , and Computation -LRB- Addison Wesley , Reading , MA -RRB- . 20 . Klein , D. & Manning , C. D. -LRB- 2004 -RRB- in Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics , eds . Daelemans , W. & Walker , M. -LRB- Barcelona -RRB- , pp. 478 485 . 21 . Klein , D. & Manning , C. D. -LRB- 2002 -RRB- in Advances in Neural Information Processing Systems , eds . Dietterich , T. G. , Becker , S. & Ghahramani , Z. -LRB- MIT Press , Cambridge , MA -RRB- , Vol . 14 , pp. 35 42 . construct generalized search path for all slots at locations j , j i 1 , , i L 2 , do i. consider all possible paths through these slots that start at vertex i and end at vertex K-L-1 ii . at each slot j compare the set of all encountered vertices to the list of existing equivalence classes , selecting the one E -LRB- j -RRB- that has the largest overlap with this set , provided it is larger than a minimum overlap -LRB- set to 0.65 in all our experiments -RRB- ; b. reduce generalized search path for each k , k i 1 , , i L 2 and all j , j i 1 , , i L 2 such that j k do i. consider the paths going through all the vertices in k that belong to E -LRB- j -RRB- -LSB- if no E -LRB- j -RRB- is assigned to a particular j , choose the vertex that appears on the original search-path at location j -RSB- ; for all j -LRB- Fig. 1F -RRB- ; ii . perform MEX on this reduced generalized path ; c. extract the leading P ; if the overlap of E -LRB- j -RRB- 1 define a new equivalence class E -LRB- j -RRB- containing only those members that did appear in the set ; d. rewire graph CREATE a new vertex corresponding to P -LRB- Fig. 1G -RRB- and replace the string of vertices subsumed by P with the new vertex P on all paths on which it occurs ; 5 . repeat step 4 -LRB- generalization -RRB- until no further significant patterns are found . We thank Alex Clark , Barb Finlay , Mark Johnson , Roni Katzir , Dan Klein , Yaniv Oshrat , Ben Sandbank , Roded Sharan , and Ed Stabler for comments ; Asa Ben-Hur and Shai Shen-Or for providing some of the bioinformatics data ; and Bo Pedersen for help with psycholinguistic testing of trained ADIOS instances . This work is partly supported by the U.S. Israel Binational Science Foundation , the Dan David Prize Foundation , and the Horowitz Center for Complexity Science . 22 . Adriaans , P. & Vervoort , M. -LRB- 2002 -RRB- in Grammatical Inference : Algorithms and Applications : 6th International Colloquium : ICGI 2002 , Lecture Notes in Computer Science , eds . Adriaans , P. , Fernau , H. & van Zaanen , M. -LRB- Springer , Heidelberg -RRB- , Vol . 2484 , pp. 293 295 . 23 . Goodman , J. T. -LRB- 2001 -RRB- A Bit of Progress in Language Modeling : Extended Version -LRB- Microsoft Research , Seattle -RRB- , Technical Report MSR-TR-2001-72 . 24 . McCandless , M. & Glass , J. -LRB- 1993 -RRB- in Proceedings of EuroSpeech ' 93 , ed . Fellbaum , K. -LRB- Berlin -RRB- , pp. 981 984 . 25 . Chelba , C. -LRB- 2001 -RRB- in Proceedings of the IEEE International Conference on Acoustics , Speech , and Signal Processing , ed . Swindlehurst , L. -LRB- IEEE , Piscataway , NJ -RRB- , Vol . 1 , pp. 544a 544d . 26 . Roark , B. -LRB- 2001 -RRB- Comput . Linguistics 27 , 249 276 . 27 . Kermorvant , C. , de la Higuera , C. & Dupont , P. -LRB- 2004 -RRB- J. Electronique d'Intelligence Artificielle , 6 . 28 . Cai , C. Z. , Han , L. Y. , Ji , Z. L. , Chen , X. & Chen , Y. Z. -LRB- 2003 -RRB- Nucleic Acids Res . 31 , 3692 3697 . 29 . Clark , A. -LRB- 2001 -RRB- Ph.D. thesis -LRB- COGS , Univ. of Sussex , Sussex , U.K. -RRB- . 30 . Wolff , J. G. -LRB- 1988 -RRB- in Categories and Processes in Language Acquisition , eds . Levy , Y , Schlesinger , I. M. & Braine , M. D. S. -LRB- Lawrence Erlbaum , Hillsdale , NJ -RRB- , pp. 179 215 . 31 . Henrichsen , P. J. -LRB- 2002 -RRB- in Proceedings of CoNLL-2002 , eds . Roth , D. & van den Bosch , A. -LRB- Assoc. Computer Linguistics , New Brunswick , NJ -RRB- , pp. 22 28 . 32 . de Marcken , C. G. -LRB- 1996 -RRB- Ph.D. thesis -LRB- MIT , Cambridge , MA -RRB- . 33 . Magerman , D. M. & Marcus , M. P. -LRB- 1990 -RRB- in Proceedings of the Eighth National Conference on Artificial Intelligence , eds . Dietterich , T. & Swartout , W. -LRB- AAAI Press , Menlo Park , CA -RRB- , pp. 984 989 . 34 . Chomsky , N. -LRB- 1986 -RRB- Knowledge of Language : Its Nature , Origin , and Use -LRB- Praeger , New York -RRB- . 35 . Elman , J. L. , Bates , E. A. , Johnson , M. H. , Karmiloff-Smith , A. , Parisi , D. & Plunkett , K. -LRB- 1996 -RRB- Rethinking Innateness : A Connectionist Perspective on Development -LRB- MIT Press , Cambridge , MA -RRB- . 36 . Nowak , M. A. , Komarova , N. L. & Niyogi , P. -LRB- 2001 -RRB- Science 291 , 114 119 . 37 . Gomez , R. L. -LRB- 2002 -RRB- Psychol . Sci . 13 , 431 436 . 38 . Fillmore , C. J. -LRB- 1985 -RRB- Berkeley Linguistic Soc . 11 , 73 86 . 39 . Goldberg , A. E. -LRB- 2003 -RRB- Trends Cognit . Sci . 7 , 219 224 . 40 . Cameron-Faulkner , T. , Lieven , E. & Tomasello , M. -LRB- 2003 -RRB- Cognit . Sci . 27 , 843 874 . 41 . Finch , S. & Chater , N. -LRB- 1991 -RRB- Artif . Intell . Simul . Behav . Q. 78 , 16 24 . 42 . Markman , E. -LRB- 1989 -RRB- Categorization and Naming in Children -LRB- MIT Press , Cambridge , MA -RRB- . 11634 www.pnas.org cgi doi 10.1073 pnas .0409746102 Solan et al. .

B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

nlp
languages
language acquisition
computational linguistics
unsupervised learning
grammar
algorithms
machine learning
statistics
learning
models
linguistics