When learning POS taggers and syntactic chunkers for low-resource languages, different resources may be available, and often all we have is a small tag dictionary, motivating type-constrained unsupervised induction. Even small dictionaries can improve the performance of unsupervised induction algorithms. This paper shows that performance can be further improved by including data that is readily available or can be easily obtained for most languages, i.e., eye-tracking, speech, or keystroke logs (or any combination thereof). We project information from all these data sources into shared spaces, in which the union of words is represented. For English unsupervised POS induction, the additional information, which is not required at test time, leads to an average error reduction on Ontonotes domains of 1.5% over systems augmented with state-of-the-art word embeddings. On Penn Treebank the best model achieves 5.4% error reduction over a word embeddings baseline. We also achieve significant improvements for syntactic chunk induction. Our analysis shows that improvements are even bigger when the available tag dictionaries are smaller. It is a core assumption in linguistics that humans have knowledge of grammar and that they use this knowledge to generate and process language. Reading, writing, and talking leave traces of this knowledge and in psycholinguistics this data is used to analyze our grammatical competencies. Psycholinguists are typically interested in falsifying a specific hypothesis about our grammatical competencies and therefore collect data with this hypothesis in mind. In NLP, we typically require big, representative corpora. NLP usually has in-Lea Frermann carried out this work while at the University of Edinburgh. duced the models from expensive corpus annotations by professional linguists, but recently, a few researchers have shown that data traces from human processing can be used directly to improve NLP models (Klerke et al., 2016; Barrett et al., 2016; Plank, 2016). In this paper, we investigate whether unsupervised POS induction and unsupervised syntactic chunking can be improved using human text processing traces. We also explore what traces are beneficial, and how they are best combined. Our work supplements psycholinguistic research by evaluating human data on larger scale than usual, but more robust unsupervised POS induction also contributes to NLP for low-resource languages for which professional annotators are hard to find, and where instead, data from native speakers can be used to augment unsupervised learning. We explore three different modalities of data reflecting human processing plus standard, pre-trained distributional word embeddings for comparison, but also because some modalities might fare better when combined with distributional vectors. Data reflecting human processing come from reading (two different eye-tracking corpora), speaking (prosody), and typing (keystroke log-ging). We test three different methods of combining the different word representations: a) canonical correlation analysis (CCA) (Faruqui and Dyer, 2014b) and b) singular value decompo-sision and inverted softmax feature projection (SVD+IS) (Smith et al., 2017) and c) simple concatenation of feature vectors. Contributions We present experiments in unsupervised POS and syntactic chunk induction using multi-modal word representations, obtained from records of reading, speaking, and writing. Individually, all modalities are known to contain syntactic processing signals, but to the best of our 2028 knowledge, we are the first to combine them in one model. Our work extends on previous work in several respects: (a) We compare using data traces from gaze, speech, and keystrokes. (b) We consider three ways of combining such information that do not require access to data from all modalities for all words. (c) While some previous work assumed access to gaze data at test time, our models do not assume access to any modalities at test time. (d) We evaluate how much the additional information helps, depending on the size of the available tag dictionary. (e) While related work on keystrokes and prosody focused on a single feature, all our word representations are multidimensional and continuous. Eye-tracking data reflect the eye movements during reading and provide millisecond-accurate records of the readers fixations. It is well established that the duration of the fixations reflect the processing load of the reader (Rayner, 1998). Words from closed word classes are usually fix-ated less often and for shorter time than words from open word classes (Rayner and Duffy, 1988). Psycholinguistics, however, is generally not interested in covering all linguistic categories, and psycholinguists typically do not study corpora, but focus instead on small suites of controlled examples in order to explore human cognition. This is in contrast with NLP. Some studies have, however, tried to bridge between psycholinguistics and NLP. Demberg and Keller (2008) found that eye movements reflected syntactic complexity . Barrett and Sgaard (2015a) and Barrett and Sgaard (2015b) have tried torespectivelypredict a full set of syntactic classes and syntactic functions across domains in supervised setups. Barrett et al. (2016), which is the work most similar to ours, used eye-tracking features from the Dundee Corpus (Kennedy et al., 2003), which has been augmented with POS tags by Barrett et al. (2015). They tried for POS induction both on token-level and type-level features. They found that eyetracking features significantly improved tagging accuracy and that type-level eye-tracking features helped more than token-level. We use the same architecture as Barrett et al. (2016). Keystroke logs also reflect the processing durations, but of writing. Pauses, burst and revisions in keystroke logs are used to investigate the cognitive process of writing (Matsuhashi, 1981; Baaijen et al., 2012). Immonen and Makisalo (2010) found that for English-Finnish translation and monolingual Finnish text production, predicate phrases are often preceded by short pauses, whereas adpositional phrases are more likely to be preceded by long pauses. Pauses preceding noun phrases grow with the length of the phrase. They suggest that the difference is explained by the processing of the predicate begins before the production of the clause starts, whereas noun phrases and adpositional phrases are processed during writing. Pre-word pauses from keystroke logs have been explored with respect to multi-word expressions (Goodkind and Rosenberg, 2015) and have also been used to aid shallow parsing (Plank, 2016) in a multi-task bi-LSTM setup. Prosodic features provide knowledge about how words are pronounced (tone, duration, voice etc.). Acoustic cues have already been used to improve unsupervised chunking (Pate and Goldwater, 2011) and parsing (Pate and Goldwater, 2013). Pate and Goldwater (2011) cluster the acoustic signal and use cluster label as a discrete feature whereas Pate and Goldwater (2013) use a quantized word duration feature. Plank (2016) and Goodkind and Rosenberg (2015) also used a single keystroke feature (keystroke pre-word pause) and the former study also discretized the feature. Our work, in contrast, uses acoustic and keystroke features as multidimensional, continuous word representations. In our experiments, we begin with five sets of word representations: prosody, keystroke, gaze as recorded in the GECO corpus, gaze as recorded in the Dundee corpus, as well as standard, text-based word embeddings from eigenwords. See below for details and references. All modalities except the pre-trained word embeddings reflect human processing of language. For all modalities, we use type-level-averaged features of lower-cased word types. The choice of using type-averaged features is motivated by Barrett et al. (2016), who tried both token-level and type-averaged eye-tracking features for POS induction and found that type-level gaze features worked better than token-level. Type-averaged features also have the advantage of not relying on access to the auxillary data at test 2029 Eigenwords Dundee Geco Keystroke Prosody n word types 46973 9109 5817 2198 598 Main P r o s od y K e ys t r o k e G e c o D undee E i gen v e c t o r O t he r 1.2 4.1 5.9 12.5 100.0 3.9 13.5 18.2 100.0 46.0 8.9 28.4 100.0 48.1 57.2 15.7 100.0 44.5 55.9 63.0 100.0 81.1 71.7 83.0 97.8 20 40 60 80 100 Figure 1 : The percentage of overlapping word types for pairs of modalities. Overlapping words are used for projecting word representations into a shared space. Read column-wise. E.g. when combining eigenwords and prosody, only 1.2% of the 46973 eigenvector word types are overlapping (bottom left), and 97.8% of the 598 prosody word types are overlapping (top right). time. Type-level averages are simply looked up in an embedding file for all previously seen words. On the other hand, type-level features obviously do not represent ambiguities, e.g., beat as a verb and a noun separately. All our features, except log-transformed word frequencies were normalized. We run unsupervised induction experiments for all ( 2 5  1 = 31 ) combinations of our five data sources on the development sets to determine which data types contribute to the task. We consider three different ways of combining modalities, two of which learn a projection into a shared space using word overlap as supervision, and one simply concatenates the embedding spaces. The combination methods are further described in  4. We list the number of word types per modality and percentage of pair-wise overlapping words in Figure 1. We only use existing data from native speaking participants, for reproducibility and in order not to get learner effects ie. biases introduced by non-native speakers. 3.2-3.5 describe each modality in detail, and how we compute the word representations. 3.1 describes a set of basic features used in all of our experiments. Like Li et al. (2012), we append a small set of basic features to all our feature sets: features relating to orthography such as capitalization, digits and suffix. Furthermore we append log word frequency and word length. Word frequencies per million are Modality n found pairs Weigh. Table 1 : Results on word association norms from wordvectors.org Correlation weighted by number of found pairs per word embedding type. obtained from British National corpus (BNC) frequency lists (Kilgarriff, 1995). Word length and word frequency explain around 70% of the variance in the eye movement (Carpenter and Just, 1983) and are therefore also important for estimating the impact of gaze features beyond such information. Plank (2016) used keystroke features for shallow chunking and did not find any benefit of normalizing word length by pre-word pause before typing each word, but Goodkind and Rosenberg (2015) did find a strong logarithmic relationship between word length and pre-word pause as well as between word frequency and pre-word pause. We use two different eye-tracking corpora. The GECO corpus (Cop et al., 2017) and the Dundee Corpus (Kennedy et al., 2003) are the two largest eye movement corpora with respect to word count. We use the native English part of the GECO corpus and the English part of the Dundee Corpus. The GECO corpus is publicly available 1 and the Dundee Corpus is available for research purposes. Participants and data The Dundee Corpus is described in Kennedy and Pynte (2005). The Dundee Corpus consists of the eye movements of 10 readers as they read the same 20 newspaper articles. For GECO, all 14 participants in the native English part read a full Agatha Christie novel. Both corpora contain > 50 . 000 words per reader. All participants for both corpora are adult, native speakers of English and skilled readers. Self-paced reading Both eye-tracking corpora reflect natural reading by making the reading self-paced and using naturally-occurring, contextual-ized text. 1 http://expsy.ugent.be/downloads/geco/ 2030 Features Eye movementslike most features reflecting human processingare very susceptible to experiment-specific effects e.g. instructions and order effects such as fatigue. Furthermore, the GECO corpus has a slightly different eye movement feature set than what we have for the Dundee corpus. Therefore we treat the two eye movement corpora as two individual modalities in order to assess their individual contributions. GECO has 34 features reflecting word-based processing. Dundee has 30 word-based features that were extracted from the raw data and previously used for POS induction by Barrett et al. (2016). For GECO, we use the features that are already extracted by the authors of the corpus. Both corpora include five word-based features e.g., first fixation duration (which is a measure said to reflect early syntactic and semantic integration), total fixation time and fixation probability. The Dundee Corpus has more features concerning the context words whereas GECO has pupil size and many features distinguishing the different passes over a word. The prosody features are described in detail in Frermann and Frank (2017) and are freely available. 2 They are derived from the Brent (Brent and Siskind, 2001) and Providence (Demuth et al., 2006) portions of the CHILDES corpus (MacWhinney, 2000), comprising longitudinal datasets of raw speech directed to 22 children, and its transcription. Word-level speech-text alignments were obtained automatically using forced alignment. For each token-level audio snippet, a set of 88 prosody features was extracted based on a previously established feature set (Eyben et al., 2016), including standard features derived from F0F3 formants, spectral shape and rhythm features, intensity and MFCC features among others. Type-level prosody features were obtained as averaged token-level features for each word type. We extracted keystroke features from the publicly available data from Killourhy and Maxion (2012). This data contains key hold times and pauses of all key presses of 20 subjects as they completed transcription and free composition tasks. We only used data from the free composition part. A pause is de-fined by the authors as the duration from keydown 2 https://github.com/ColiLea/prosodyAOA to keydown. The free composition data consists of a total of 14890 typed words and 2198 word types. For each word, we extracted the following features: (i) average key hold duration of all characters associated with producing the word, (ii) pre-word pause, (iii) hold duration of space key before word, (iv) pause length of space key press pause before word, and (v) ratio of keypresses used in the word production to length of the final word. For each word, we also included these five features for up to 3 words before. In total, we have 5  4 = 20 keystroke features. We use lower-cased word type averages, as with the other modalities. Eigenwords are standard, pre-trained word embeddings, induced using spectral-learning techniques (Dhillon et al., 2015). We used the 30-dimensional, pre-trained eigenvectors. 3 3.6 Preliminary evaluation Our application of these word representations and their combinations is unsupervised POS and syntactic chunk induction, but before presenting our projection methods in  4 and our experiments in  5, we present a preliminary evaluation of the different modalities using word association norms. Table 1 shows the weighted correlation between cosine distances in the representations and the human ratings in the word association norm datasets available at wordvectors.org (Faruqui and Dyer, 2014a). Eigenwords, not surprisingly, correlates better than the representation based on processing data  with the exception of prosody. The correlation with prosody is non-significant, however, because of the small sample size. We now have word representations from different, complementary modalities, with very different coverages, but all including a small overlap. We assume that the different modalities contain complementary human text processing traces because they reflect different cognitive processes, which motivates us to combine these different sources of information. Our assumption is con-firmed in the evaluation. The fact that we have very low coverage for some modalities, and the 3 http://www.cis.upenn.edu/ungar/ eigenwords/ 2031 fact that we have an overlap between all our vocabularies, specifically motivates an approach, in which we use the intersection of word types to learn a projection from two or more of these modalities into a shared space. Obviously, we can also simply concatenate our representations, but because of the low coverage of some modalities and because co-projecting modalities has some regularization effect, we hypothesize that it is better to learn a projection into a shared space. This hypothesis is verified by the results in  6. The simplest way of combining the modalities is concatenating the corresponding vectors for each word. The different modalities have different di-mensionalities, so we would need to perform dimensionality reduction to sum or average vectors, and the non-overlapping words don't allow for e.g. taking the outer product, so we simply concatenate the vectors instead. We use 0 for missing values. 4.2 and  4.3 describe two different projection methods for projecting the representations in the different modalities into a shared space. We use the intersection of the lower-cased vocabulary for the alignment, i.e., as a supervision signal. For example, if the words man , dog and speak exist in both eigenword and keystroke data, from these 2 x 3 vectors, CCA estimate the transformation for the vectors for house , cat and boy , which (in this example) only exists in the keystroke data. Canonical Correlation Analysis (CCA), as originally proposed by Hotelling (1936), is a method of finding the optimum linear combination between two sets of variables, so the set of variables are transformed onto a projected space while the correlation is maximized. We use the implementation of Faruqui and Dyer (2014b) made for creating bilingual embeddings. We use modalities instead of languages. The size of the projected space is smaller than or equal to the original dimension. We incrementally combine modalities and project them to new, shared spaces using the intersection of the lower-cased vocabulary. We add them by the order of word type count starting with the modality with most word types. For the first projection only, we reduce the size of the projected space. We set the ratio of the first projected space (only two modalities) to 0.6 based on POS induction results on development data using the setup described in  5. As an alternative to CCA, but closely related, we also use a projection method proposed and implemented by Smith et al. (2017), which uses singular value decomposition and inverted softmax (SVD+IS). This method uses a reference space, rather than projecting all modalities into a new space. Smith et al. (2017) apply SVD+IS to obtain an orthogonal transformation matrix that maps the source language into the target language. In addition, in order to estimate their confidence on the predicted target, they use an inverted softmax function for determining the probability that a target word translates back into a source word. Like for CCA, we incrementally project datasets onto each other starting with the most word-type rich modality. We use the highest dimensionality of any of our representations (88 di-mensions). This section presents our POS and syntactic chunk induction experiments. We present the datasets we used in our experiments, the sequence tagging architecture, based on second-order hidden Markov models, as well as the dictionary we used to constrain inference at training and test time. For unsupervised POS induction, we use Ontonotes 5.0 (Weischedel et al., 2013) for training, development and test. We set all hyper-parameters on the newswire ( NW ) domain, optimizing performance on the development set. Size of the development set is 154,146 tokens. We run individual experiments on each of the seven domains, with these hyper-parameters, reporting performance on the relevant test set. The domains are broadcast conversation ( BC ), broadcast news ( BN ), magazines ( MZ ), newswire ( NW ), the Bible ( PT ), telephone conversations ( TC ), and weblogs ( WB ). We also train and test unsupervised POS induction on the CoNLL 2007 (Nivre et al., 2007) splits of the Penn Treebank (Marcus et al., 1993) using the hyper-parameter settings from Ontonotes. We mapped all POS labels to Google's coarse-grained, universal POS tagset (Petrov et al., 2012). For model selection, we select based both on best results on Ontonotes 2032 Rules DET  NP VERB  VP NOUN | PRONOUN | NUM  NP . Table 2 : Heuristics for expanding our POS dictionary to chunks NW development as well as Penn Treebank development sets. For syntactic chunk induction, we use the bracketing data from Penn Treebank with the standard splits for syntactic chunking. We tune hyperpa-rameters for chunking on the development set and select best models based on the development result. We used a modification of the implementation of a type-constrained, second-order hidden Markov model with maximum entropy emissions from Li et al. (2012) (SHMM-ME). It is a second-order version of the first order maximum entropy HMM presented in (Berg-Kirkpatrick et al., 2010) with the important addition that it is constrained by a crowd-sourced tag dictionary (Wiktionary). This means that for all words in the Wiktionary, the model is only allowed to predict one of the tags listed for it in Wiktionary The same model was used in Barrett et al. (2016) to improve unsupervised POS inducing using gaze data from the Dundee Corpus, and in Bin-gel et al. (2016) to augment an unsupervised POS tagger with features from fMRI recordings. The number of EM iterations used for inducing our taggers was tuned using eigenvector embeddings on the development data, considering values 1..50. PoS performance peaked at iterations 30 and 31. We use 30 in all our POS experiments. For syntactic chunking, we use 48 iterations, which led to the best performance on the PTB development data using only eigenword embeddings. The Wiktionary constrains the predicted tags in our model. The better the Wiktionary, the better the predictions. For POS-tagging we used the same Wiktionary Feature set TA No embeddings 60.32 Eigenwords 59.26 Best combined models CCA Dun GECO Pros 63.33 *  SVD+IS GECO Key Pros 62.91* Concat Eig GECO Key 61.16 Table 3 : Chunk tagging accuracy. Best models from CCA, SVD+IS and concatenation. Model section on development set. * p < . 001 Mcnemar midp test when comparing to no embeddings. p < . 001 Mcnemar midp test when comparing to Eigenwords.) dump 4 that Li et al. (2012) used in their original experiments. The Wiktionary dump associated word types with Google's universal parts-of-speech labels. For chunking, Wiktionary does not provide direct information about the possible labels of words. We instead apply simple heuristics to relate POS information to syntactic chunking labels. Since we already know the relation between words and POS labels from Wiktionary, we can compute the transitive closure in order to obtain a dictionary relating words with syntactic chunking labels. We present the heuristics in Table 2. Note that the rules are rather simple. We do not claim this is the best possible mapping. We are relying on these simple heuristics only to show that it is possible to learn syntactic chunkers in an unsupervised fashion by relying on a combination of features from different modalities and a standard, crowd-sourced dictionary. All our POS tagging accuracies can be seen in Table 4. Our first observation is that human processing data helps unsupervised POS induction. In fact, the models augmented with processing data are consistently better than the baseline without vector representations, as well as better than only using distributional word embeddings. Generally, CCA seems to find the best projection into a common space for system combinations. For Penn Treebank, the CCA-aligned model is the best and this result is significant ( p < 4 https://code.google.com/archive/p/wikily-supervised-pos-tagger/ 2033 Ontonotes PTB Feature set BC BN MZ NW PT TC WB avg No embeddings 83.1 84.41 85.32 84.94 85.14 77.8 85.93 83.81 82.83 Eigenwords 83.16 84.68* 85.48 85.07 85.31 78.07 85.88 83.95 83.38* Best Ontonotes NW models CCA Eig Dun 83.45 *  84.99 * 85.79* 85.38 *  85.2 77.99 86.38 *  84.17 84.28 *  SVD+IS Dun GECO Key 83.24 84.76 86.22 *  85.33*  85.44 77.84 85.95 84.11 84.25*  Concat Eig Dun GECO 83.39*  84.78* 85.8*  85.36*  85.45 78.38 * 86.21  84.19 83.91*  Best PTB models CCA Eig Dun 83.45 *  84.99 * 85.79* 85.38 *  85.2 77.99 86.38 *  84.17 84.28 *  SVD+IS Dun Key 83.24 84.59 86.12*  85.28*  85.39 77.90 85.86 84.05 84.24*  Concat Eig Pros 83.22 84.54 85.67 85.01 84.98 77.98 85.97 83.91 84.22*  Table 4 : POS tagging accuracies for baselines and the model combinations that performed best on newswire development data ( NW ). Best performance per domain is boldfaced. *) p < . 001 McNemar midp test when compared to the no embeddings condition for the corresponding test set. ) p < . 001 McNemar midp test when compared to eigenwords for the corresponding test set. . 001 ) when comparing both to no embeddings and eigenwords. For Ontonotes 5.0, CCA is better than the other projection methods in 4/7 domains, but when averaging, concatenation gets the higher result. The standard embeddings are often part of the best combinations, but the human processing data contributes with important information; in 4/7 domains as well as on PTB data, we see a significantly better performance ( p < . 001 ) with a combination of modalities when comparing to eigenwords. Aligning Dundee with eigenwords is the best POS model both according to the Ontonotes 5.0 NW development set and the Penn Treebank development set. Dundee is the most frequent modality in the six best POS induction models with five appearances. Eigenwords is second most frequent with four appearances. The syntactic chunking accuracies are in Table 3. Also here CCA is the better combination method. For chunking, all combined models are better than no embeddings and eigenwords. The improvement is significant compared to no embeddings for concatenation p < . 001 . For CCA, the result is significantly better than no embeddings and eigenwords. For chunking, GECO data appears in all best models and is thus the most frequent modalities. Keystroke and prosody appears in two best models each. Table 5 : Graph similarities in [0 ,  ) , 0 = identical. Nearest neighbor graphs We include a detailed analysis of subgraphs of the nearest neighbor graphs in the embedding spaces of keystrokes, Dundee, GECO, and CCA projection of all modalities. Specifically, we consider the nearest neighbor graphs among the 15 most frequent unam-bigous nouns, according to Wiktionary. 5 See Figure 2 for plots of the nearest neighbor graphs. The prosody features containing less than 600 word types only contained 2 of the 15 nouns and is therefore not included in this analysis. Projecting word representations into a shared space using linear methods assumes approximate isomorphism between the embedding spaces or at least their nearest neighbor graphs. We use the VF2 algorithm (Cordella et al., 2001) to verify that the subgraphs are not isomorphic, but this can also be seen directly from Figure 2. Neither keystroke and gaze embeddings, nor the two different gaze-induced embeddings are isomorphic. 5 Wiktionary is a crowd-sourced, imperfect dictionary, and one of the unambiguous nouns is spends , which, we assume, you are more likely to encounter as a verb. modalities Figure 2 : Nearest neighbor graphs for 15 frequent nouns. Since none of the modalities induce isomorphic nearest neighbor graphs, this does not tell us much about similarities between modalities. To quantify the similarity of non-isomorphic graphs, we use eigenvector similarity Shigehalli and Shet-tar (2011), which we calculate by computing the Laplacian eigenvalues for the nearest neighbors, and for each graph, find the smallest k such that the sum of the k largest eigenvalues is < 90% of the eigenvalues. We then take the smallest k of the two, and use the sum of the squared differences between the largest k eigenvalues as our similarity metric. Using this metric to quantify graph similarity, we see in Table 5 that, not surprisingly, the gaze graphs are the most similar. The projected space is more similar to the gaze spaces, but balances gaze and keystroke information. The GECO embeddings agree more with the keystrokes than the Dundee embeddings does. t-SNE plots We take words thataccording to the Wiktionarycan only have one tag and sort them by BNC frequency (Kilgarriff, 1995) in descending order. For these words and their POS tags we get the feature vector of the POS model yielding the highest result on both Ontonotes and PTB: CCA-projected eigenwords and Dundee features. For the first 200 occurrences of the frequency-sorted list, we reduce dimensionality using t-Distributed Stochastic Neighbor Embedding (t-SNE) (Maaten and Hinton, 2008) and plot the result. Figure 3 shows that 200 most frequent content words cluster with respect to their POS tag, somewhat distinguishing verbs from nouns and adjectives from adverbs in CCA space. Our Wiktionary for English contains POS information for 72,817 word types. Word types have 6.2 possible POS categories on average meaning we have over 450.000 entries in our POS dictionary. For Penn Treebank, 70.0% of wordtypes of the test set are covered by the dictionary. For the chunking data, 70.4% of wordtypes of the test set are covered by the dictionary. The English Wiktionary is thus much bigger than wik-tionaries for low-resource language (Garrette and Baldridge, 2013). How big a dictionary is needed to achieve good performance, and can we get away with a smaller dictionary if we have processing data? This section explores the performance of the model as a function of the Wiktionary size. We sorted the Wiktionary by word frequency obtained from BNC (Kilgarriff, 1995) and increased the Wiktionary size for the best POS system starting with 0 (no dictionary). For each Wiktionary size, we compare with the baseline without access to processing data and eigenwords. The learning curve can be seen in Figure 4a and Figure 4b. We observe that having entries for the most frequent words is a lot better than having no dictionary, and that the difference between our best system and the baseline exists across all dictionary sizes. With 10,000 entries, all systems seems to reach a plateau. Genres and domains When collecting our human language processing data, we did not control for genre. Our data sets span child-directed speech, free text composition, and skilled adults reading fiction and newspaper articles. The 2035 20 10 0 10 20 30 20 10 0 10 20 NOUN VERB Figure 3 : t-SNE plots of CCA-projected eigen dundee features for pairs of tags. (b) 0-150,000 entries Figure 4 : Learning curve assuming Wiktionary entries for k most frequent words, comparing our best PoS induction system against our baseline. On Ontonotes WB development data, 30 training iterations. Dundee corpus (newspaper articles) matches the genre of at least some of the Ontonotes test set. Immonen and Makisalo (2010) found that for keystroke, genre does seem to have an effect on average pause length, be it sentence initial, word initial, clause initial or phrase initial. Texts organized linearlye.g. reports and narrativesrequire less pausing than texts with a global approach, like expository, persuading and generalizing text. Our results show that human processing features transfer across genres, but within-genre data would probably be beneficial for results. Richer representations The type-level features we use, do not take context into account, and the datasets we use, are too small to enrich our representations. Human processing data is more and more readily available, however. Eye trackers are probably built into the next generation of consumer hardware, and speech records and keystroke logs are recordable with existing technology. We have shown how to improve unsupervised POS induction and syntactic chunking significantly using data reflecting human language processing. Our model, which is a second-order hidden Markov model, is the first to combine multidimensional, continuous features of eye movements, prosody and keystroke logs. We have shown that these features can be combined using projection techniques, even when they only partially overlap in word coverage. None of our models require access to these features at test time. We experimented with all combinations of modalities, and our results indicate that eye tracking is useful for both chunking and POS induction. Finally, we have shown that the potential impact of human processing data also applies in a low-resource setting, i.e., when available tag dictionaries are small. Thanks to Desmond Elliott for valuable ideas. This research was partially funded by the ERC Starting Grant LOWLANDS No. 313695, as well as by Trygfonden.
abstain abstain result abstain abstain abstain result result abstain method method abstain abstain abstain objective objective objective objective abstain method abstain abstain abstain objective abstain objective method method method method method other other other other other other other other other other other method other other other other other other other other other other objective abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain result method abstain abstain result objective result abstain result result other other