Another possible metric for optimization would be to weigh each reference instance equally, wherein there is no need to calculate Independent Accuracy for each entity and then average them into Adaptation Accuracy, as shown earlier.

such an approach will lead the learning agent to ignore the entities that are least referred to, and focus on getting the reference to the most frequently referred-to entities right.


Note that as the value of n increases from 1, accuracy increases as it provides more evidence for classification.

after a certain point the adaptation accuracy started to stabilize, because too much sensing is not more informative.


Therefore, when there is no verbal feedback (i.e., no clarification request) from the user, the system has no information on which a user profile can be picked.

the learned policy represents this uncertainty in its state transistions and is able to select an appropriate adaptive action.


The difference between the Learned-DS and the Jargon-adapt policy is statistically significant (p < 0.05).

the difference between the Learned-DS and the Stereotype policy is not significant.


That would require Arg1 to be "there isn't likely to be any silver lining."

the annotators did not take such an argument to be minimal.


r Multiplicity: The PDTB allows more than one sense label to be associated with a single discourse connective to indicate that multiple sense relations hold concurrently (e.g., a token of since may be labeled with both a temporal and causal sense).

propBank only permits a constituent to fill a single functional role.


They propose a user model for jointly generating keywords and questions.

their approach is based on generating question templates from existing questions, which requires a large set of English questions as training data.


its relevance to the topic in consideration.

q3 of the state-of-the-art was assigned a lower score due to its lack of clarity with respect to the topic.


We can regard the task of sentiment lexicon learning as word-level sentiment classification.

for wordlevel sentiment classification, it is not straightforward to extract features for a single word.


It appears that the antonym relations depict word relations in a more accurate way and can refine the word sentiment scores more precisely.

the synonym relation and word alignment relation dominate, whereas the antonym relation accounts for only a small percentage of the graph.


This parsing algorithm is defined for featureless variants of TAG.

in implemented TAGs (e.g., XTAG [The XTAG Research Group 2001], SemXTAG [Gardent 2008], or XXTAG 1 [Alahverdzhieva 2008]) feature structures and feature unification are central.


(Independent derivation) because they may involve strong scopal and morpho-syntactic constraints, stacked predicative verbs (i.e., verbs taking a sentential complement, Example (5a)) and non-intersective modifiers (Example (5c)) require dependent derivations.


To ensure the appropriate linearization, the Schabes and Shieber's approach introduces the outermost-predication rule, which stipulates that predicative trees adjoin above modifier auxiliary trees.

the FB-TAG approach allows both orders and lets feature constraints rule out ungrammatical sentences such as Example (14b).


The success of DSMs in essentially word-based tasks such as thesaurus extraction and construction (Grefenstette 1994;Curran 2004) invites an investigation into how DSMs can be applied to NLP and information retrieval (IR) tasks revolving around larger units of text, using semantic representations for phrases, sentences, or documents, constructed from lemma vectors.

the problem of compositionality in DSMs-of how to go from word to sentence and beyond-has proved to be non-trivial.


Its properties are well known, and it becomes simple to evaluate the meaning of a sentence if given a logical model and domain, as well as verify whether or not one sentence entails another according to the rules of logical consequence and deduction.

such logical analysis says nothing about the closeness in meaning or topic of expressions beyond their truth-conditions and which models satisfy these truth conditions.


In this model, nouns are lexical vectors, as with other models.

embracing a view of adjectives that is more in line with formal semantics than with distributional semantics, they model adjectives as linear maps taking lexical vectors as input and producing lexical vectors as output.


Whereas the approaches to compositional DSMs presented in Section 2 either failed to take syntax into account during composition, or did so at the cost of not being able to compare sentences of different structure in a common space, this categorical approach projects all sentences into a common sentence space where they can be directly compared.

this alone does not give us a compositional DSM.


The general learning algorithm presented in Section 4.3 technically can be applied to learn and model relations of any semantic type.

many open questions remain, such as how to deal with logical words, determiners, and quantification, and how to reconcile the different semantic types used for sentences with transitive and intransitive sentences.


This information is reported in their paper as additional means for model comparison.

for the same reason we considered Spearman's ρ to be a fair means of model comparisonnamely, in that it required no model score normalization procedure and thus was less likely to introduce error by adding such a degree of freedom-we consider the HIGH/LOW means to be inadequate grounds for comparison, precisely because it requires normalized model scores for comparison to be meaningful.


They also more generally demonstrated that concrete models could be built from the general categorical framework and perform adequately in simple paraphrase detection tasks.

various aspects of compositionality were not evaluated here.


The creation of treebanks is a prime example (Marcus, Santorini, and Marcinkiewicz 1993).

the linguistic theories motivating these annotation efforts are often heavily debated, and as a result there often exist multiple corpora for the same task with vastly different and incompatible annotation philosophies.


For dependency parsing, an effective guiding feature is the dependency path between the hypothetic head and modifier, as shown in Figure 3.

our effort is not limited to this, and more special features are introduced: A classification label or dependency path is attached to each feature of the baseline classifier to generate combined guiding features.


It would also be valuable to evaluate the improved word segmenter and dependency parser on the out-of-domain data sets.

currently most corpora for word segmentation and dependency parsing do not explicitly distinguish the domains of their data sections, making such evaluations difficult to conduct.


Eschewing theorizing to stay close to data permits a remarkably wide range of linguistic phenomena to be covered, and it is this that is the book's greatest strength.

in a few places, a seemingly arbitrary theoretical perspective is assumed rather more tacitly than one might hope, with few hints as to alternative analyses (e.g., see the following remarks about parts of speech in Chapter 6).


Reordering of the German word stimmen is internal to the phrase-pair gegen ihre Kampagne stimmen -'vote against your campaign' and therefore represented by the translation model.

the model fails to correctly translate the test sentence shown in Figure 1(b), which is translated as 'they would for the legalization of abortion in Canada vote', failing to displace the verb.


The model makes no phrasal independence assumption and generates a tuple monotonically by looking at a context of n previous tuples, thus capturing context across phrasal boundaries.

n-gram-based systems have the following drawbacks.


The POS-based rewrite rules serve to precompute the orderings that will be hypothesized during decoding.

notice that this rule cannot generalize to the test sentence in Figure 1(b), even though the tuple translation model learned the trigram < sie -'they' würden -'would' stimmen -'vote' > and it is likely that the monolingual language model has seen the trigram they would vote.


Note that the OSM, like the discontinuous phrase-based model (Galley and Manning 2010), allows all possible geometries as shown in Figure 7.

because our decoder only uses continuous phrases, we cannot hypothesize (ii) and (iii) unless they appear inside of a phrase.


Our model, like the reordering models (Tillmann and Zhang 2005;Galley and Manning 2008) used in phrase-based decoders, is lexicalized.

our model has richer conditioning as it considers both translation and reordering context across phrasal boundaries.


As already mentioned in Example 6, the grammar G constructed there is not prefix-closed.

we can make it prefixclosed by explicitly allowing the "missing" rule instances: We shall now argue that this modification does not actually change the language generated by G .


Construction of H. The chain of inclusions Y ⊆ L(H ) ⊆ L(G) is sufficient to prove Lemma 6: Because Y and L(G) are Parikh-equivalent (which we observed at the beginning of Section 3.4.2), so are L(H ) and L(G), which means that L(H ) satisfies all of the properties claimed in Lemma 6, even though this does not suffice to prove our current lemma.

once H is given, it is not hard to also obtain a grammar H that generates exactly Y.


In the grammar formalisms folklore, the generative capacity of CCG is often attributed to generalized composition, and indeed we have seen (in Lemma 4) that even grammars without target restrictions can generate non-context-free languages such as L(G 2 ).

our results show that composition by itself is not enough to achieve weak equivalence with TAG: The yields of the transformed derivations from Section 3.4 form a context-free language despite the fact that these derivations may still contain compositions, including compositions of degree n > 2.


Perhaps surprisingly, it is not the availability of generalized composition rules by itself that explains the generative power of CCG, but the ability to constrain the interaction between generalized composition and function application by means of target restrictions.

one may be interested in CCG primarily as a formalism for developing grammars for natural languages (Steedman 2000;Baldridge 2002;Steedman 2012).


From this point of view, the suitability of CCG for the development of lexicalized grammars has been amply demonstrated.

our technical results still serve as important reminders that extra care must be taken to avoid overgeneration when designing a grammar.


A simple answer is to add some lexicalized method for enforcing target restrictions to CCG, specifically on the application rules.

we are not aware that this idea has seen widespread use in the CCG literature, so it may not be called for empirically.


We note that this is not always the case: For example, the entailment graph in Figure 3 is not an FRG, because X annex Y entails both Y be part of X and X invade Y, while the latter two do not entail one another.

we hypothesize that this scenario is rather uncommon.


Assuming that entailment graphs are FRGs allows us to use the node re-attachment operation in linear time.

this assumption also enables performing other graph operations efficiently.


Comparing runtimes for TNF and GNF, we see that the gap between the algorithms decreases as −λ increases.

for reasonable values of λ, TNF is about four to seven times faster than GNF, and we were unable to run GNF on large graphs, as we report in Section 5.


As expected, HTL-FRG is much faster than both TNF and TNCF, and TNCF is somewhat slower than TNF.

as mentioned earlier, TNCF is able to improve both precision and recall compared with TNF and HTL-FRG.


Ideally, we need examples of sentences annotated with polarity for the whole sentence as well as sentiment tags for constituents within a sentence, as with the Penn TreeBank for training traditional linguistic parsers.

this is not practical as the annotations will be inevitably time-consuming and require laborious human efforts.


Specifically, Figure 10a indicates that a minimum fragment frequency that is too small will introduce noise, and it is difficult to estimate reliable polarity probabilities for infrequent fragments.

a minimum fragment frequency that is too large will discard too much useful information.


The results become better as the beam size K increases.


As shown in the results, the polarity probabilities learned by our method are more reasonable and meet people's intuitions.

there are also some negative examples caused by "false subjective."


They define an aggregate evaluation score for comparing systems, estimating expected value and standard error for hypothesis testing.

in aggregating this way information about ties is lost.


Similarly, in the natural language processing field (NLP), the adaptation of a generic model to a specific domain often requires new annotated data that illustrate its specificities (as in Candito, Anguiano, and Seddah 2011).

the creation cost of such data highly depends on the kind of labels used to adapt the model.


If π is greater or equal to 1, the refining procedure fixes the whole best solution and the residual problem is then empty.

π is set to π min if a better solution is found in order to challenge G and improve this solution.


Because the recording and the post-recording annotation process are expensive tasks, the recording length of such a corpus has to be as short as possible.

in order to train a domain-specific dependency parser, the covering of POS sequences may be useful for increasing the diversity of syntax patterns.


This seems to contradict the idea that the performance of LamSCP improves as the search area becomes wider.

the distributions of the units to cover in Gutenberg and Le-Monde are different, and the variation on the length of the sentences in Le-Monde is very high, which may account for this slight difference in terms of gain.


As previously observed for both algorithms, their computation times grow when the number of required covering features increases.

the ratio between the computation time of LamSCP and ASA does not behave as in Experiment 2 (see Table 7): For the k-covering of 1-POS, this ratio increases from 290 (±11) to 657 (± 43) when k goes from 1 to 5, and for the k-covering of 2-POS, it increases from 75 (±5) to 108 (±6).


Sporleder and Lascarides (2005) included other features (e.g., words and their stems, Part-of-Speech [POS] tags, positions, segment lengths) in a boosting-based classifier (i.e., BoosTexter [Schapire and Singer 2000]) to further improve relation classification accuracy.

these studies evaluated classification performance on the instances where rhetorical relations were originally signaled (i.e., the discourse cues were artificially removed), and did not verify how well this approach performs on the instances that are not originally signaled.


That approach has the advantages of making the parsing process easier, and the model gets more data to learn from.

for a solution like ours, which tries to capture the interdependencies between constituents, this would be problematic with respect to scalability and inappropriate because of two modeling issues.


Similarly, the relative distributions of Background, Contrast, Cause, and Explanation are different in the two parsing scenarios.

different kinds of features are applicable and informative for intraversus multi-sentential parsing.


Algorithm 1 describes how CODRA generates the unit sequences at different levels of the candidate DTs for a given number of EDUs in a sentence.

specifically, to compute the probability of a DT constituent in doing so, it may generate some duplicate sequences.


Discourse cues (e.g., because, but), when present, signal rhetorical relations between two text segments, and have been used as a primary source of information in earlier studies (Knott and Dale 1994;Marcu 2000a).


For example, the root of the rhetorical sub-tree spanning over EDUs e 1:2 in Figure 9b is Elaboration-NS.

extraction of these features assumes the presence of labels for the sub-trees, which is not the case when we apply the parser to a new text (sentence or document) in order to build its DT in a nongreedy fashion.


In addition to these features, we also experimented with other features including WordNet-based lexical semantics, subjectivity, and TF.IDF-based cosine similarity.

because such features did not improve parsing performance on our development set, they were excluded from our final set of features.


For example, we observe that over 12% of the sentences in the instructional corpus of Subba and Di-Eugenio (2009) have leaky boundaries.

we notice that in most cases where DT structures violate sentence boundaries, its units are merged with the units of its adjacent sentences, as in Figure 13b.


If the unlexicalized version is also found to be rare, other variations of the production, depending on whether they include the lexical heads and how many non-terminals (one or two) they consider before and after the potential boundary, are examined one after another (see Fisher and Roark [2007] for details).

we compute the maximum likelihood estimates for a primary production (feature) and its other variations, and use those directly as features with/without binarizing the values.


In RST-DT, by our count, 7, 321 out of 7, 673 sentences in the training set, 951 out of 991 sentences in the test set, and 1, 114 out of 1, 208 sentences in the doubly-annotated set have a well-formed DT.

3, 032 out of 3, 430 sentences in the instructional corpus have a well-formed DT.


This demonstrates the potential of TSP SW for data sets with even more leaky boundaries, e.g., the Dutch (Vliet and Redeker 2011) and the German Potsdam (Stede 2004) corpora.

it would be interesting to see how other heuristics to do consolidation in the cross condition (Section 4.3.


Some other measures, namely, all α from Krippendorff, and the new γ introduced in this article, are computed from observed and expected disagreements (instead of agreements), denoted here respectively D o and D e , and they define the final agreement by Equation (2).

a o − a e 1 − a e (1) the way the expected value is computed is the only difference between many coefficients (κ, S, π, and their generalizations), and is a controversial question.


In zone (1) of the left part of Figure 4, one annotator has created two units (of the same category), and the other annotator has created only one unit covering the same space.

once the continua are discretized, the two annotators seem to agree on this zone (with the four same atoms), as we can see in the right part of the figure .


Three examples of split and permutation are shown in the right part of the figure, for split positions of, respectively, 15, 24, and 38, all coming from the same real continuum, with units that are no longer aligned (except by chance).

we have to address the fact that some units may intersect with themselves, generating some part of agreement beyond chance.


A consequence of this approach is illustrated in Figure 15, where two annotators perfectly agree both on positions and categories in the experiment on the left, and still perfectly agree on position but slightly diverge concerning categories in the experiment on the right (1/2, 6/7, and 8/9 are assumed to be close categories).

u α drops from 1 in the left experiment to -0.34 (a negative value means worse than random) in the right experiment, despite, in the latter, the positions being all correct, and the categories being quite good, since c|u α = 0.85.


In the current version of the CST, the false positive error type creates some overlapping (new units may overlap), and it is the reason why u α and κ d were discarded from this experiment.

we have kept c|u α because it behaves quite well despite overlapping units.


Admittedly, c|u α was not designed to handle these configurations (and so should not be included in this experiment), but surprisingly it   seems to perform in rather the same way as it does with no overlapping; this must be investigated further, but judging from this preliminary observation, it seems this coefficient could still be operational and useful in such cases.

u α does not handle correctly this experiment and so was not included in the graph.


Due to the inherent difficulty in obtaining the true value of h * from a text, however, these arguments are based only on indirect clues with respect to convergence.

hilberg conjectured a decrease in the human conditional entropy, as follows (hilberg 1990): he obtained this through an examination of Shannon's original experimental data and suggested that β ≈ 0.5.


For corpus RongoA-c, we consider a character inclusive of all adjoining parts (i.e., including accents and ornamental parts).

for corpus RongoB-c, we separate parts as reasonably as possible, among multiple possible separation methods.


Whether a value for h * is reached asymptotically and also whether h * > 0 remain important questions requiring separate, more extensive mathematical and empirical studies.

h 2 (or Yule's K, bottom graphs) showed convergence, already at the level of 10 5 tokens, for both words and characters.


Given this result, it is doubtful that V is convergent across languages.

h 1 is mathematically proven to be convergent given infinite-length randomized data, but to larger values than those of the original texts, as mentioned in Section 3.2.


Typically, statistical machine translation (SMT) systems (Chiang 2007;Koehn 2010) perform generation into the target language as part of an integrated system, which avoids the high computational complexity of independent word ordering.

performing word ordering separately in a pipeline has many potential advantages.


In our previous papers (Zhang and Clark 2011;Zhang, Blackwood, and Clark 2012), we applied a set of beams to this structure, which makes it similar to the data structure used for phrase-based MT decoding (Koehn 2010).

we will show later that this structure is unnecessary when the model has more discriminative power, and a conceptually simpler single beam can be used.


For CKY decoding, the model is used to compare hypotheses within each chart cell, which cover the same input words.

for the best-first search decoder, the model is used to order hypotheses on the agenda, which can cover different numbers of words.


In our previous papers we observed empirical convergence of online learning using this linear model, and obtained competitive results.

as explained in Section 2, only positive examples were expanded during training, and the expansion of negative examples led to non-convergence and made online training infeasible.


In both the first and second case, a gold-standard edge is pruned as the result of the expansion of a negative example.

in order for the gold-standard goal edge to be constructed, all gold-standard edges that have been expanded must remain in the chart.


However, SEARN is more oriented to greedy search, optimizing local decisions.

our framework is oriented to best-first search, optimizing global structures.


In recent years, many studies have been published on data collected from social media, especially microblogs such as Twitter.

rather few of these studies have considered evaluation methodologies that take into account the statistically dependent nature of such data, which breaks the theoretical conditions for using cross-validation.


This is what most previous studies have done.


Recent years have seen a growing interest in computational modeling of metaphor, with many new statistical techniques opening routes for improving system accuracy and robustness.

the lack of a common task definition, shared data set, and evaluation strategy makes the methods hard to compare, and thus hampers our progress as a community in this area.


These include, most notably, the comparison view, formulated in the Structure-Mapping Theory of Gentner (1983), and the interaction view (Black 1962;Hesse 1966).

the principles of CMT have inspired and influenced much of the computational work on metaphor, thus becoming more central to this paper.


The answer to this question most likely depends on the NLP application in mind.

generally speaking, real-world NLP applications are unlikely to be concerned with historical aspects of metaphor, but rather with the identification of figurative language that needs to be interpreted differently from the literal language.


This means it needs to be either data-driven and be able to automatically acquire the knowledge it needs from text corpora, or rely only on large-scale, general-domain lexical resources (that are already in existence and do not need to be created in a costly manner).

it would be an advantage if no such resource is required and the system can dynamically induce meanings in context.


The resource has been criticized for the lack of clear structuring principles of the mapping ontology (Lönneker-Rodman 2008).

to date MML is the most comprehensive resource for conceptual metaphor in the linguistic literature, and the examples from the list have been used by computational approaches (Mason 2004;Krishnakumaran and Zhu 2007;Li, Zhu, and Wang 2013), both for development and evaluation purposes.


The improvement over Turney's evaluation set-up was the annotation of complete sentences rather than isolated phrases.

it should be noted that the system was evaluated on selected examples rather than continuous text.


They evaluated the method using 10-fold cross-validation and report an F-score of 0.75.

they did not evaluate their system on metaphorical language independently.


The results are encouraging and show that porting coarse-grained semantic knowledge across languages is feasible.

it should be noted that the generalization to coarse semantic features inevitably only captures shallow behavior of metaphorical expressions in the data and bypasses conceptual information.


The authors presented some interesting examples of conceptual metaphors the system extracted, which they claim may foster critical thinking in social science.

they did not carry out any quantitative evaluation.


Shutova (2010) tested her system only on metaphors expressed by a verb and reports an accuracy of 0.81, as evaluated on top-ranked paraphrases produced by the system.

she used WordNet for supervision, which limits the number and range of paraphrases that can be identified by her method.


On one hand, such violations are indicative of any kind of non-literalness (i.e., not only metaphor, but also, for instance, metonymy) or anomaly in language and the approach is likely to overgenerate.

in the case of most conventional metaphors that are highly frequent, no statistically significant violation can be detected in the data, and the approach would bypass many such metaphors.


Such a technique attained a precision of 0.17 and a recall of 0.55, suggesting that the selectional preference violation hypothesis does not port well beyond handcrafted descriptions to large-scale, data-driven techniques.

other, "non-violation" applications of selectional preferences have been fruitful in metaphor modeling.


Exploiting the wider topical structure of text is a promising avenue for metaphor processing.

one needs to keep in mind that distributional similaritybased methods risk assigning frequent metaphors to target domains (as is the case for other semantic violation-based methods).


Based on the results of these experiments, concreteness is likely to be a practically useful feature for metaphor processing.

it should be noted that Turney's hypothesis (that target words tend to be abstract and source words tend to be concrete) explains only a fraction of metaphors and does not always hold.


Another benefit of this type of evaluation is that it allows one to assess both the precision and the recall of the system.

only two of the presented approaches (Dunn 2013b; Shutova 2013) conducted this type of evaluation, as shown in Table 7.


That is, mentioning Iraq in the second sentence is not necessary (for a human being) to understand the meaning of the text.

making both references explicit, as shown in Example 2, would be redundant and could lead to the perception that the text is merely a concatenation of two independent sentences -rather than a set of adjacent sentences that form a meaningful, or coherent, discourse.


Each of these tasks focuses, however, on a different level of linguistic analysis from ours: Following the definitions embraced by Recasens and Vila (2010), "paraphrasing" is a relation between two lexical units that have the same meaning, whereas "coreference" indicates that two referential expressions point to the same referent in discourse.

3 to work on paraphrasing, we are specifically interested in pairs of text fragments that involve implicit arguments, which can only be resolved in context.


The difficulty of this task can also be seen in the results for the Greedy model, which only achieves an F 1 -score of 17.2%.

we observe that the majority of all sure alignments can be retrieved by applying the LemmaId model (60.3% recall).


There are two main reasons for this: On the one hand, their model makes use of much larger resources to compute alignments, including a paraphrasing database that contains over 7 million rewriting rules; on the other hand, their model is supervised and makes use of additional data to learn weights for each of their features.

full and EMNLP'12 only make use of a small development data set to determine a threshold for graph construction.


We found that annotators made use of the full rating scale, with the extremes indicating either a strong preference for the text on the left-hand side or the righthand side, respectively.

most ratings were concentrated more towards the center of the scale (i.e., around zero).


2009; Levy and Goldberg 2014) that models learning from input informed by dependency parsing, rather than simple running-text input, yield improved similarity estimation and, specifically, clearer distinction between similarity and association.

we find no evidence for a related hypothesis (Agirre et al.


Using the output of these two models as input to a logistic regression classifier, Turney predicts whether two concepts are associated, similar, or both, with 61% accuracy.

in the absence of a gold standard covering the full range of similarity ratings (rather than a list of pairs identified as being similar or not) Turney cannot confirm directly that the similarityfocused model does indeed effectively quantify similarity.


We have established the validity of similarity as a notion understood by human raters and distinct from association.

much theoretical semantics focuses on relations between words or concepts that are finer-grained than similarity and association.


Hyper/hyponym pairs that are separated by fewer levels in the WordNet hierarchy are both more strongly associated and rated as more similar.

there are also interesting discrepancies between similarity and association.


(2012) model to focus on association rather than similarity.

the true explanation may be less simple, since the Huang et al.


While the NLM is the strongest performer on WS-353, SVD is the strongest performer on MEN.

the NLM model performs notably better than the alternatives at modeling similarity, as measured by SimLex-999.


This aligns with the (also unexpected) observation that humans rate the similarity of adjectives more consistently and with more agreement than other parts of speech (see the dashed lines).

the parallels between human raters and the models do not extend to verbs and nouns; verb similarity is rated more consistently  than noun similarity by humans, but models estimate these ratings more accurately for nouns than for verbs.


In particular, for models to learn high-quality representations for all linguistic concepts, we believe that future work must uncover ways to explicitly or implicitly infer "deeper," more general, conceptual properties such as intentionality, polarity, subjectivity, or concreteness (Gershman and Dyer 2014).

although improving corpusbased models in this direction is certainly realistic, models that learn exclusively via the linguistic modality may never reach human-level performance on evaluations such as SimLex-999.


2013) or GloVe (Pennington, Socher, and Manning 2014).

this is not actually building Deep Learning models, and I hope in the future that more people focus on the strongly linguistic question of whether we can build meaning composition functions in Deep Learning systems.


Shortly after the release of Chinese-English and English-Chinese translation services, we also released translation services between Chinese and Japanese, Korean, and other daily-used foreign languages.

with translation directions expanded, users' expectations for the translation between the resource-poor languages became higher and higher.


There are also researchers who regard query reformulation as the translation from the original query to the rewritten one (Riezler and Liu 2010).

what interests me the most is the encounter between translation technology and Chinese traditional culture.


Other people all thought it was impossible and laughed at him.

yugong said to the people calmly: "Even if I die, I have children; and my children would have children in the future.


Machine translation (MT) has long been both one of the most promising applications of natural language processing technology and one of the most elusive.

over approximately the past decade, huge gains in translation accuracy have been achieved (Graham et al.


With regard to the first distinction, local features, such as phrase translation probabilities, do not require additional contexts from other partial derivations, and they are computed independently from one another.

when features for a particular phrase pair or synchronous rule cannot be computed independently from other pairs, they are called non-local features.


Dense features are generally easier to optimize, both from a computational point of view because the smaller number of features reduces computational and memory requirements, and because the smaller number of parameters reduces the risk of overfitting.

sparse features allow for more flexibility, as their parameters can be directly optimized to increase translation accuracy, so if optimization is performed well they have the potential to greatly increase translation accuracy.


More formally, we can cast the problem as minimizing the expectation of (•), or risk minimization:ŵ Here, Pr(F, E) is the true joint distribution over all sets of input and output sentences that we are likely to be required to translate.

in reality we will not know the true distribution over all sets of sentences a user may ask us to translate.


Intuitively, if λ is set to a small value, optimization will attempt to learn a w that effectively minimizes loss on the training data, but there is a risk of overfitting reducing generalization capability.

if λ is set to a larger value, optimization will be less aggressive in minimizing loss on the training data, reducing over-fitting, but also possibly failing to capture useful information that could be used to improve accuracy.


Converting this to a loss function that is dependent on the model parameters, we obtain the following loss expressing the error over the 1-best results obtained by decoding in Equation 1: Error has the advantage of being simple, easy to explain, and directly related to translation performance, and these features make it perhaps the most commonly used loss in current machine translation systems.

it also has a large disadvantage in that the loss function expressed in Equation 17is not convex, and most MT evaluation measures used in the calculation of the error function error(•) are not continuously differentiable.


If this error function can be composed as the sum of sentence-level errors, such as BLEU+1, choosing the oracle is simple; we simply need to find the set of candidates that have the lowest error independently sentence by sentence.

6 1: procedure ORACLE( F, E, C ) 2: 3: repeat 5: if s < s then Update the oracle 11: else if s = s then Same error value 14: return O 20: end procedure when using a corpus-level error function we need a slightly more sophisticated method, such as the greedy method of Venugopal and Vogel (2005).


In standard approaches to batch learning, for every training example f (i) , e (i) we enumerate every translation and derivation in the respective sets E ( f (i) ) and D( f (i) ), and attempt to adjust the parameters so we can achieve the translations with the lowest error for the entire data.

as mentioned previously, the entire space of derivations is too large to handle in practice.


One of the major advantages of online methods is that updates are performed on a much more fine-grained basis-it is often the case that online methods converge faster than batch methods, particularly on larger data sets.

online methods have the disadvantage of being harder to implement (they often must be implemented inside the decoder, whereas batch methods can be separate), and also generally being less stable (with sensitivity to the order in which the training data is processed or other factors).


We then decode each source sentencẽ f ( j) of the mini-batch and generate a k-best list (line 7), which is used in optimization (line 9).

to the batch learning algorithm in Figure 3, we do not merge the k-bests from previous iterations.


In the entirety of this article, we have assumed that optimization for MT aims to reduce MT error defined using an evaluation measure, generally BLEU.

as mentioned in Section 2.5, evaluation of MT is an active research field, and there are many alternatives in addition to BLEU.


From these statistics we can see that even after over ten years, MERT is still the dominant optimization algorithm.

starting in WMT 2013, we can see a move to systems based on MIRA, and to a lesser extent ranking, particularly in the most competitive systems.


The fact that algorithms other than MERT are seeing adoption in competitive systems for shared tasks is a welcome sign for the future of MT optimization research.

there are still many open questions in the field, a few of which can be outlined here: Stable Training with Millions of Features: At the moment, there is still no stable training recipe that has been widely proven to effectively optimize millions of features.


From a theoretical point of view, it can be seen as incorporating discriminative training techniques when working with a generative model by optimizing for segmentation performance rather than maximum a posteriori probability.

only the hyperparameters are optimized in this fashion, whereas the lexicon parameters are still learned within the generative model framework.


Because acquiring the posteriors analytically is intractable, inference is performed utilizing Markov chain Monte Carlo algorithms to obtain samples from the posterior distributions of interest (Johnson 2008;Johnson and Goldwater 2009;Johnson and Demuth 2010;Sirts and Goldwater 2013).

as sampling-based models are costly to train on large amounts of data, we adopt the parsing-based method proposed in Sirts and Goldwater (2013) to use the trained AG model inductively on test data.


Both generative and discriminative models can be extended to utilize annotated as well as unannotated data in a semi-supervised manner.


The log-linear model presented by Poon, Cherry, and Toutanova (2009) is omitted because it does not have a freely available implementation.

the model has been compared in the semi-supervised learning setting on Arabic and Hebrew with CRFs and Morfessor previously by Ruokolainen et al.


Interpreting precision and recall requires some care as it is always possible to reduce over-segmentation errors by segmenting less and, conversely, to reduce undersegmentation errors by segmenting more.

if this is taken into account, the error categorization can be quite informative.


These two grammar versions have no difference when trained transductively.

when training an inductive model, it may be beneficial to store the subtrees corresponding to whole words because these trees can be used to parse the words in the test set that were seen during training with a single rule.


Morfessor appears to favor precision over recall (see Finnish) in the event a trade-off takes place.

the AG heavily favors recall (see English).


9 Intuitively, WORDS should yield zero recall.

when applying macro averaging, a word having a gold standard analysis with no boundaries yields a zero denominator and is therefore undefined.


Meanwhile, the second extension AG SELECT (SSV) also results in overall higher precision by reducing over-segmentation of STEM segments substantiallyalthough, for Finnish, SUFFIX is oversegmented compared with AG SELECT (USV).

whereas both AG (SSV) and AG SELECT (SSV) improve recall on Finnish compared to AG (USV), only AG (SSV) succeeds in improving recall for English.


This result again supports the intuition that in order to learn the open categories, one is required to utilize large amounts of word forms for learning.

it appears that the necessary information can be extracted from unannotated word forms.


This approach was taken by Poon, Cherry, and Toutanova (2009), Spiegler and Flach (2010), and Sirts and Goldwater (2013).

as discussed in Section 3.3.1, for the Morfessor family the fixing approach was outperformed by the weighted objective function (Kohonen, Virpioja, and Lagus 2010).


The online learning algorithms that discard each new training sample after updating the learner are also referred to as incremental learning algorithms by some authors (see Anthony and Biggs [1992]).

this constraint can be relaxed by using mini-batches (small sets of samples).


The RRR measure reflects how frequently unseen n-grams are repeated in the corpus to be translated.

such unseen n-grams constitute only a fraction of the in-domain corpus.


Plots were obtained for the XRCE and Europarl training corpora and the three translation directions (from English to Spanish, French, and German).

in the figure only the XRCE English to Spanish (Figure 2a) and the Europarl English to Spanish (Figure 2b) results are reported 7 (very similar results were obtained for the other language pairs).


It is expected that updating the system in a sentence-wise manner will produce the best results.

this updating strategy poses efficiency problems because of the necessity of executing model updates in real time.


EM convergence experiments provided in Section 4.4 showed that the log-likelihood of HMM-based word alignment models using the incremental version of the EM algorithm is competitive with that obtained by using the conventional version.

it is still unclear if the use of online learning will cause a degradation in the quality of the translations with respect to the use of batch learning.


Hopefully, this modification will allow the system to provide more accurate predictions for similar samples.

modifying parameters may also produce lateral effects.


As seen in Table 6, online learning allowed us to obtain around one point of improvement in the three measures under consideration with respect to the conventional system (without retraining).

the improvements were not statistically significant in some cases (WER for English to French, BLEU and WER for English to German).


Online learning has been a main topic of research in the field of machine learning.

in the SMT framework, the vast majority of the work has been devoted to the study of the batch-learning setting.


As we can see, interlaced updates increased the training time with respect to that of basic online updates (this was the expected outcome since five samples were processed at each trial instead of one).

the time costs of interlaced updates were still affordable for both corpora (worst case times of a few seconds, median times less than 1 second for the different values of R).


This example suggests a simple division of reordering patterns into long range, or global, and short range, or local.

other language pairs display more complex, hierarchical patterns.


This happens indirectly, through the scoring of target word n-grams, which are generated by translating the source positions in different orders.

the fixed-size context of language models used in SMT (typically four or five words) makes them largely insensitive to global reordering phenomena.


This quantity can be interpreted as the number of times that a human reader would have to "jump" between words to recover the correct translation order.

no distinction is made between short and long-range reordering errors.


(2011) introduce yet another reordering-specific metric, called fuzzy reordering score (FRS) which, like the KRS, is independent from lexical choice and measures the similarity between a sentence's reference reordering and the reordering produced by an SMT system (or by a pre-ordering technique).

whereas Birch, Osborne, and Blunsom (2010) used Kendall's tau between the two sentence permutations, Talbot et al.


As also noted by Fox (2002), this kind of reordering is not strictly necessary to produce accurate and fluent translations, but its occurrence in parallel corpora affects the automatic reordering measures.

a qualitative analysis can profit from the extensive work done by linguists and grammaticians to abstract the fundamental properties of a language.


(2008) and Birch, Blunsom, and Osborne (2009), PSMT performs similarly or better than HSMT for the Arabic-to-English language pair.

hSMT was shown to better cope with the reordering of VSO sentences (Bisazza 2013).


French and Arabic [ Main order: different; CDiff: 1.5; PDiff: 1 ] At the clause level, this pair differs in main word order (SVO versus VSO or SVO) like the English-Arabic pair, but also in the order of negation and verb.

phrase-level order is notably more similar, with only one discordant feature of minor importance (adjective and degree word).


More specifically, for a fixed CFG, we can solve the recognition problem in time cubic in the length of the input string, using standard dynamic programming techniques.

for a fixed SCFG we can still recognize the input pair [u, v] in polynomial time, but the degree of the polynomial depends on the specific structure of the synchronous rules of the grammar, and can be much larger than in the CFG case.


Chart parsing can be generalized to SCFGs.

in the latter case parse trees no longer generate single substrings: They rather generate tuples with several substrings.


There are recent attempts to address some of these issues by using alternative characterizations of word meaning that do not involve creating a partition of usages into senses (McCarthy and Navigli 2009;Erk, McCarthy, and Gaylord 2013), and by asking WSI systems to produce soft or graded clusterings (Jurgens and Klapaftis 2013) where tokens can belong to a mixture of the clusters.

these approaches do not overtly consider the location of a lemma on the continuum, but doing so should help in determining an appropriate representation.


AIC depends on the sample size (through p(Y|M)), so in order to be able to compare all models that model the same partitionability estimate, we compute AIC only on the subset of lemmas that enters in all analyses.

22 we compute the F test on all lemmas where the clusterability measure is valid, 23 in order to use the largest possible set of lemmas to test the viability of a model.


For example, our previous work Ng 2009, 2012) experimented with various techniques for combining a small bitext for a resource-poor language (Indonesian or Spanish) with a much larger bitext for a related resource-rich language (Malay or Portuguese), pretending that Spanish is resource-poor; the target language of all bitexts was English.

that work did not attempt language adaptation, except for very simple transliteration for Portuguese-Spanish that ignored context entirely; because it does not substitute a word with a completely different word, transliteration did not help much for Malay-Indonesian, which use unified spelling.


The target sentence is shown in T. Because the decoders used in SMT and ASR typically work at the phrase-or the word-level, they cannot make use of sentence-level features.

our text rewriting decoder works at the sentence-level, that is, all hypotheses are complete sentences.


We believe this is because lattice encodes many options, but does not use a Malay language model, and 1-best uses a Malay language model, but has to commit to 1-best.

cN:word uses both n-best outputs and an Indonesian language model.


For instance, for the tweet Yoona taking the ' ' (be healthy)ˆˆ, the segment (be healthy would not be valid, because it does not contain the parenthesis closer ).

both (be healthy) and be healthy are acceptable.


Thus, if these are the only characters that are aligned during the computation of the translation score, both languages are equally likely to be correct.

the Hiragana characters , , and that surround it only exist in Japanese, so the entire sequence is much more likely to be Japanese than Chinese.


Once P L (x | l) is computed for all possible values of l and x, calculating the language score can be trivially computed.

computing the translation score φ T requires the computation of the word alignments in φ T over all possible segmentations, which requires O(|x| 6 ) operations using a naive approach [O(|x| 4 ) segmentation candidates, and O(|x| 2 ) operations for aligning each pair of segments].


it is desirable to apply λ +u whenever possible, because it only requires one operation.


If we are looking for a particular language pair, this is fine, but when we search for translations in a large number of language pairs, this is impractical.

in most cases, many language pairs can be trivially ruled out.


The previous section describes a method for finding the most likely segments p, q, u, v and the language pair l, r and the alignments a, according to the IDA model score, for any document x.

extracting parallel sentences using this model requires addressing other issues, such as identifying the tweets that contain translations from those that do not.


Section 4.3 presented a supervised method to train a classifier that discriminates parallel and non-parallel data.

training requires annotated instances where parallel and non-parallel segments are identified.


We will rely on crowdsourcing to produce annotated data sets, which has been successfully used to generate parallel data (Ambati and Vogel 2010; Zaidan and Callison-Burch 2011; Post, Callison-Burch, and Osborne 2012; Ambati, Vogel, and Carbonell 2012).

these methods have been focused on using workers to translate documents.


For instance, the ratio between parallel and non-parallel tweets for Arabic-English in Twitter is 2:1 in the annotated data sets, which is definitely not the case in a uniformly extracted data set.

6 performing the annotation in a uniformly extracted data set is problematic for two reasons.


In some cases, these improvements can also be the result of overfitting.

a strong indication that this is not the case is the fact that similar results can be obtained for the Twitter test set, where we can observe a BLEU improvement from 9.55, using the NIST corpus, to 23.57, using the Weibo corpus.


As one could expect, shifted addition is on average closer to actual PMI values than plain addition.

weighted addition provides better approximations to the observed values.


The constructive proofs presented in Section 2.3 and 2.4 define two kinds of oracles.

they are not directly applicable when the transition combination strategy is utilized.


For any graph, we can call this algorithm and get a corresponding tree.

the tree is informative only when the given graph is dense enough.


The objective of training a symbol-refined bigram tagger is to solve the LA-involved emission and transition parameters by maximizing the likelihood of the training data.

with a non-symbol-refined HMM tagger, where the POS tags are observed, the latent annotations are unseen variables.


A natural strategy for extending current experiments is to include both clustering results together.


Note that the overall tagging performance of the Berkeley parser is significantly worse than the sequence models.

better POS tagging does not lead to better parsing.


The LCFRS shown here does not satisfy our normal form requiring each rule to have either two nonterminals on the right-hand side with no terminals in the composition function, or zero nonterminals with a composition function returning fixed strings of terminals.

it can be converted to such a form through a process analogous to converting a CFG to Chomsky Normal Form.


As long as we move the first index from row to column, rather than from column to row, the intermediate results will require addresses no longer than the length of i.

if ϕ(B) = d, then every configuration in which B appears is balanced: If ϕ(B) = d and B appears in more than one configuration, that is, |config(B)| > 1, it is impossible to copy entries for B between the cells using a matrix of size (2n) d .


In order to optimize the complexity of our algorithm, we want to minimize d, which is the maximum over all rules A → B C of For a fixed binarized grammar, d is always less than p, the tabular parsing complexity, and, hence, the optimal d * over binarizations of an LCFRS is always less than the optimal p * for tabular parsing.

whether any savings can be achieved with our algorithm depends on whether ωd * < p * , or ωd * + 1 < p * in the case of balanced grammars.


Subsequent readability research by Crossley, Greenfield, and McNamara (2008) looked only at content overlap and showed it to be a significant feature.

similar work by Pitler and Nenkova (2008) did not lead to the same conclusion.


(2010) found that enlarging the corpus, which exclusively consisted of texts for primary school children, with more diverse text material allowed for an overall better performance.

the added value of the discourse relations to the system was still not significant.


We can conclude that the introduction of more complex linguistic features has indeed proven useful.

the discussion on which features are the best predictors remains open.


For the PoS-related features, we observe a clear difference between the English and Dutch data sets in that 78% of the English features versus only 48% of the Dutch features correlate (i.e., 21 versus 13 out of 27 to be exact).

for both languages at least one feature representing the five main part-of-speech classes (nouns, adjectives, verbs, adverbs, and prepositions) does correlate.


Having a general characterbased approach that works well across languages could be useful, for example, in a streamlined intelligence application that is required to work efficiently on a wide range of languages for which NLP tools or language experts might not be readily available, or where the user does not desire a complex customization for each language.

kernels based on a different kind of information (for example, syntactic and semantic information) can be combined with string kernels via MKL to improve accuracy in some specific situations.


The empirical results shown in Table 5 indicate that the presence bits kernel and the kernel based on LRD obtain better results on the raw text documents.

the results of the intersection kernel are roughly the same, or only slightly better than, the results presented in Table 4.


(2012), whereas KRR based on the same kernel reaches an accuracy of only 82.3%, which is 2.3 percentage points lower than the state-of-the-art (84.6%).

kRR overturns the results when the two kernels are combined through MkL.


Diving into details, we can see that the results obtained by KRR are higher than those obtained by KDA.

both methods perform very well compared with the state-of-the-art.


The translation function deals with this by postponing this decision with the help of λ-bound formulas representing roles.

when we start translating a fresh AMR, we start with the root node.


Speakers may thus make use of specific words or stylistic elements to represent themselves in a certain way.

because of this agency, social variables cease to have an essential connection with language use.


Sociolinguists must then also select an appropriate methodology.

typical methods used within sociolinguistics would require sampling the data down.


For example, based on data from Twitter (a popular microblogging site) dialectal variation has been mapped using a fraction of the time, costs, and effort that was needed in traditional studies (Doyle 2014).

data from CMC are not always easy to collect.


Furthermore, although historically the field of sociolinguistics started with a major focus on phonological variation (e.g., Labov 1966), the use of social media data has led to a higher focus on lexical variation in computational sociolinguistics.

there are concerns that a focus on lexical variation without regard to other aspects may threaten the validity of conclusions.


2014recruited participants using e-mail, social media, and blogs, which resulted in a sample that was likely to be biased towards linguistically interested people.

they did not expect that the possible bias in the data influenced the findings much.


Sociolinguistic studies have found that adolescents use the most nonstandard forms, because at a young age the group pressure to not conform to established societal conventions is the largest (Eckert 1997;Holmes 2013).

adults are found to use the most standard language, because for them social advancement matters and they use standard language to be taken seriously (Eckert 1997;Bell 2013).


We discussed variables such as gender, age, and geographical location, thereby mostly focusing on the influence of social structures on language use.

as we also pointed out, speaker agency enables violations of conventional language patterns.


Furthermore, most studies within computational linguistics generally assume that texts are written in one language.

these assumptions may not hold, especially in social media.


To make use of the rich repertoire of theory and practice from sociolinguistics and to contribute to it, we have to appreciate the methodologies that underlie sociolinguistic research, for example, the rules of engagement for joining in the ongoing scientific discourse.

as we have highlighted in the methodology discussion (Section 2), the differences in values between the communities can be perceived as a divide.


Sociolinguistic studies typically control for multiple social variables (e.g., gender, age, social class, ethnicity).

many studies in computational sociolinguistics focus on individual variables (e.g., only gender, or only age), which can be explained by the focus on social media data.


A Formal Distributional Semantics thus holds the promise of developing a more comprehensive model of meaning.

given the fundamentally different natures of FS and DS, building an integrative framework poses theoretical and engineering challenges.


Some approaches to composition can actually be seen as sitting between the F-first and D-first approaches: In Coecke, Sadrzadeh, and Clark (2011) and Grefenstette and Sadrzadeh (2011), a CCG grammar is converted into a tensor-based logic relying on the direct composition of distributional representations.

with their F-first counterparts, D-first FDSs regard distributions as the primary building blocks of the sentence, which must undergo composition rules to get at the meaning of longer constituents (see above).


FDS promises to give us a much better coverage of natural language than either Formal or Distributional Semantics.

much remains to be done; here we address some prominent limitations of current approaches and propose directions for future research.


This would capture the intuition we spelled out in the introduction that a wolf is a better non-dog than a screwdriver.

the proposal of Hermann and colleagues is, again, purely theoretical, and we do not see how domain and value features with the desired properties could be induced from corpora on a large scale.


As usually assumed, the alternative set for a sentence is the set of semantic values resulting from replacing the negated element with arbitrary values of the right semantic type.

it is clear that not all alternatives are created equal.


13 Supervision alone has virtually no effect on performance, and composition alone has a strong negative effect.

by combining composition and supervision, we obtain an increase in correlation of about 1% for both IT and THERE.


We recognize that the discussion in this conclusion is very speculative in nature, and much empirical and theoretical work remains to be done.

we hope to have demonstrated that accounting for negation, far from being one of the weak points of this formalism, is one of the most exciting directions in the development of a fully linguistically motivated theory of distributional semantics.


In that task, methods were tested on their ability to match a term (in this case a verb) with its definition (usually of the form verb-object), such as embark: enter boat or vessel.

the definitions of Kartsaklis, Sadrzadeh, and Pulman were mined from a set of dictionaries, whereas the RELPRON relative clauses are not limited to dictionary definitions.


Because there is an intuitive definition-like "flavor" to some relative clauses, we considered using dictionary definitions as a source of relative clauses for the data set.

in practice we found that short, natural definitions in relative clause format are rare in dictionaries.


The relative clauses in RELPRON are restrictive relative clauses, because they narrow down the meaning of the head noun.

a non-restrictive relative clause provides incidental situational information; for example, a device, which was in the room yesterday.


Despite this fact, there is a large literature investigating how such operators can be used for phrasal composition, starting with the work of Lapata (2008, 2010) (M&L subsequently).

the additive model from M&L has the general form: where u and v are word (column) vectors in R n (e.g., − → fast and − → car), p ∈ R n is the vector for the phrase resulting from composition of u and v ( − −−− → fast car), and A ∈ R n×n and B ∈ R n×n are matrices that determine the contribution of u and v to p. M&L make the simplifying assumption that only the ith components of u and v contribute to the ith component of p, which yields the form: the parameters α, β ∈ R allow the contributions of u and v to be weighted differently, providing a minimal level of syntax-awareness to the model.


We find that the scores are relatively well-balanced across head nouns, but that more concrete terms and their properties may be easier to model.

based on qualitative observations, a more important factor seems to be term polysemy.


This may be a feature of the topic domain, in that the activities undertaken by different kinds of sports players-golfer, batter, pitcher, quarterback, and so forth-are distributionally similar.

all methods exhibit only average ability to identify the correct head noun for organization terms, but relatively high ability to select the correct organization properties when the head noun is known.


person properties, such as person that defends rationalism (philosopher).

19 when restricted to ranking organization properties, as in Table 10, SPLF achieves a perfect MAP of 1.0 for religion.


Dinu and Lapata (2010) advocate a method that resembles LVW, in that it uses a distribution over latent dimensions in order to measure semantic shifts in context.

whereas their approach computes the contextualized meaning directly within the latent space, the LVW approach we adopt in this article exploits the latent space to determine the features that are important for a particular context, and adapt the original (out-of-context) dependency-based feature vector of the target word accordingly.


McNally and Boleda (2016) offer empirical and conceptual arguments in favor of the TCL dual approach to meaning and, like us, see DS as an ally in specifying the internal content aspect of composition.

we offer a much more detailed and specific investigation of the interactions between TCL and particular methods of DS composition.


Following Baroni and Lenci (2010), we use typed dependency relations as the bases for our distributional features, and following Padó and Lapata (2007), we include higher-order dependency relations in this space.

in contrast to previous proposals, the higher-order dependency relations provides structure to the space that is crucial to our definition of composition.


Because we permit paths that traverse both forwards and backwards along the same dependency-for example, in the co-occurrence white/JJ, AMOD •AMOD, dry/JJit is logical to consider white/JJ, AMOD • DOBJ •DOBJ•AMOD, dry/JJ a valid co-occurrence.

in line with our decision to include white/JJ, , white/JJ rather than white/JJ, AMOD •AMOD, white/JJ , all co-occurrence types are canonicalized through a dependency cancellation process in which adjacent, complementary dependencies are cancelled out.


For example, many of the dimensions that make sense for verbs, such as those involving a co-occurrence type that begins with DOBJ or NSUBJ, do not make sense for a noun.

as we now explain, the co-occurrence type structure present in an APT allows us to address this, making way for our definition of distributional composition.


In particular, Baroni and Lenci showed that typed co-occurrences based on grammatical relations were better than untyped cooccurrences for distinguishing certain semantic relations.

as shown by Weeds, Weir, and Reffin (2014), it does not make sense to compose typed features based on firstorder dependency relations through multiplication and addition, because the vector spaces for different parts of speech are largely non-overlapping.


For example, in a lorry carries apples, there is a path of length 2 between the nouns lorry and apples via the node carry.

they also used a word-based basis mapping, which essentially reduces all of the salient grammatical paths to untyped co-occurrences.


Hence the phrases happiest blonde person and blonde happiest person receive the same dependency representation and therefore also the same semantic representation.

we believe that our approach is flexible enough to be able to accommodate a more sensitive grammar formalism that does allow for distinctions in modifier scope to be made if an application demands it.


It does not seem particularly useful at this point to speculate about phenomena that either a distributional approach or a logic-based approach would not be able to handle in principle, as both frameworks are continually evolving.

logical and distributional approaches clearly differ in the strengths that they currently possess (Coecke, Sadrzadeh, and Clark 2011;Garrette, Erk, and Mooney 2011;Baroni, Bernardi, and Zamparelli 2014).


It is a software package that contains implementations of a variety of MLN inference and learning algorithms.

developing a scalable, general-purpose, accurate inference method for complex MLNs is an open problem.


The knowledge bases used are WordNet and PPDB.

with our work, PPDB paraphrases are not translated to logical rules (Section 5.3).


We first notice that the phrasal subset is generally harder than the lexical subset: none of the feature sets on their own provide dramatic improvements over the baseline, or come particularly close to the ceiling score.

using all features together does better than any of the feature groups by themselves, indicating again that the feature groups are highly complementary.


For this reason, and because crossing dependencies have traditionally been rare in corpora of languages like English, Chinese, or Japanese, many implementations of dependency parsers assume projectivity (Nivre 2006).

crossing dependencies are needed to represent some linguistic phenomena like topicalization, scrambling, wh-movement, or extraposition, so it is necessary for natural language parsers to support non-projectivity, especially when working with languages with flexible word order.


It is clear that the tasks involved in preprocessing content (normalization, POS tagging, and parsing) are clearly in the realm of natural language processing, and so are tasks such as named entity recognition, opinion mining, and event extraction.

the book does not do a good job of distinguishing what new techniques are required due to distinguishing properties of the social media domain.


⋆ The car dealer should sell trucks, provide sports cars, be located in France.

many syntactic and linear ordering choices are regulated by soft constraints-that is, yield text of variable acceptability.


Moreover, taking individual decisions at different sub-tasks in a sequential manner might lead to suboptimal solutions (Marciniak and Strube 2005).

symbolic joint approaches to microplanning lack in robustness and efficiency.


In the course of distributional clustering, concrete concepts (e.g., water, coffee, beer, liquid) tend to be clustered together when they have similar meanings.

abstract concepts (e.g., marriage, democracy, cooperation) tend to be clustered together when they are metaphorically associated with the same source domain(s) (e.g., both marriage and democracy can be viewed as mechanisms or games).


2013;Tsvetkov, Mukomel, and Gershman 2013), as well as our own approach, share this intuition.

our methods are different in their aims.


Unfortunately, solving this minimization problem is NP hard (Wagner and Wagner 1993;Von Luxburg 2007).

an approximate solution can be found by relaxing the constraints on the elements of H in constraint 4.


The tt constraints are designed to reinforce this principle.

introducing the st type of constraints allows us to investigate to what extent explicitly reinforcing the source domain features in clustering allows to harvest more target domains associated with the source.


The TT constraints are designed to reinforce this principle.

introducing the TS type of constraint allows us to investigate to what extent explicitly reinforcing the source domain features in clustering allows us to harvest more target domains associated with the source.


Incorporating new seed expressions is thus likely to increase the recall of the system without a considerable loss in precision.

creating seed sets for new languages may not always be practical.


Another important reason AGG fails is that it by definition organizes all concepts into a tree and optimizes its solution locally, taking into account a small number of clusters at a time.

being able to discover connections between more distant domains and optimizing globally over all concepts is crucial for metaphor identification.


The performance of AGG in the identification of metaphorical expressions is higher than in the identification of metaphorical associations, because it outputs only few expressions for the incorrect associations.

wN tagged a large number of literal expressions due to the incorrect prior identification of the underlying associations.


The differences in performance across languages are mainly explained by the differences in the quality of the data and pre-processing tools available for them.

both our quantitative results and the analysis of the system output confirm that all systems successfully discover metaphorical patterns from distributional information.


The use of annotated metaphorical mappings for supervision at the clustering stage does not significantly alter the performance of the system, because their patterns are already to a certain extent encoded in the data and can be learned.

metaphorical expressions are a good starting point in learning metaphorical generalizations in conjunction with clustering techniques.


Evaluation was performed in terms of measuring the acceptance of the "main argument" using the automatically recognized entailments, yielding an F 1 score of about 0.75.

to our work, which deals with micro-level argumentation, Dung's model is an abstract framework intended to model dialogical argumentation.


We observed that they contain many artificial controversies or non-sense topics (for instance, createdebate.com) or their content is professionally curated (idebate.org, for example).

we admit that debate portals might be a valuable resource in the argumentation mining research.


The role of backing is to give additional support to the warrant, but there is no warrant in our model anymore.

what we observed during the analysis was the presence of some additional evidence.


We also observed documents in our data that were purely sarcastic (the pathos dimension); therefore logical analysis of the argument (the logos dimension) would make no sense.

given the structure of such documents, some claims or premises might also be identified.


2010) (In some cases, inclusion can work fantastically well., For the majority of the children in the school, mainstream would not have been a suitable placement.).

most claims that are used for instance in the prayer in schools arguments are very direct, without trying to diminish its commitment to the conveyed belief (for example, NO PRAYER IN SCHOOLS!...


All the topics except private vs. public schools exhibit similar amounts of verifiable non-experiential premises (9% to 22%), usually referring to expert studies or facts.

this type of premise has usually the lowest frequency.


In principle, one document can contain multiple independent arguments.

29 only 4% of the documents in our data set contain arguments for both sides of the issue.


single token does not convey enough information that could be encoded as features for a machine learner.

as discussed in Section 4.4.5, the annotations were performed on data pre-segmented to sentences and annotating tokens was necessary only when the sentence segmentation was wrong or one sentence contained multiple argument components.


Second, as compared to the negative experiences with annotating using Walton's schemes (see Sections 4.4.1 and 3.1), our modified Toulmin model offers a trade-off between its expressiveness and annotation reliability.

we found that the capabilities of the model to capture argumentation depend on the register and topic, the length of the document, and inherently on the literary devices and structures used for expressing argumentation as these properties influence the agreement among annotators.


Ruppenhofer and Rehbein (2012) argue that a frame-based representation of evaluative language is suitable for capturing multi-word evaluative expressions and idioms such as give away the store and sentiment composition.

apart from using semantic frames for identifying the topics (or targets) of sentiment (Kim and Hovy 2006) and deriving an intensity-based sentiment lexicon (Raksha et al.


(2013) also use a specific scheme of discourse relations.

rather than relying on gold discourse annotations, they jointly predict sentiment, aspect, and discourse relations and show that the model improves accuracy of both aspect and sentiment polarity at the sub-sentential level.


This might concern framing relations like BACKGROUND and CIRCUMSTANCE but also some temporal relations such as SEQUENCE or SYNCHRONOUS.

relations can be grouped according to their similar effects on both subjectivity and polarity analysis.


For example, earthquake and election are considered bursty.

non-bursty words are those that appear more consistently throughout documents discussing different topics-use and they, for example.


(2008) present results on three language pairs (English-Spanish, English-Chinese, and English-Arabic).

evaluation is only done over nouns, which is a bursty word class, and lexicons are limited to high-frequency words.


Another option, taken by Goldberg and Nivre, is to follow the one-best action predicted by the classifier.

initial experiments showed that the one-best approach did not work well.


Most transliteration systems are trained on a list of transliteration pairs which consist of a word and its transliteration.

manually labeled transliteration pairs are only available for a few language pairs.


The only model parameter that cannot be estimated on the labeled training data is the interpolation parameter λ.

the test data consists of both transliterations and non-transliterations.


One possible solution in the current set-up could be to convert Chinese to Pinyin and then apply transliteration mining.

we did not try it under the scope of this work.


However, the system achieves more balanced precision and recall and a higher F-measure than the unsupervised system.

to the unsupervised and semi-supervised systems, the supervised transliteration mining system estimates the posterior probability of non-transliteration λ on the test data.


1 Qualitatively, certain patterns are evident; for example, there is high similarity between TIBs 8, 9, and 10 (metalinguistic or unrelated speech) and between TIBs 2 and 3 (different requests for confirmation) in Dementia.

the latter pair are not as similar among Controls, perhaps because of a different degree of reduction in TIB 2.


Our pilot work involved focusing only on TIBs in the CCC database, which involves free-form conversation.

without a task to complete, the POMDP consistently decided that the best way to avoid confusion is to simply say nothing at all; therefore, we only consider the DementiaBank data here, as described subsequently.


Instead, our task is to identify the central concept the question is testing (e.g., measuring speed), and to eliminate words that are part of an example or narrative (e.g., turtle, path) that are unlikely to contribute much utility (or, may introduce noise) to the QA process.

because at a high level focus words identify the information need of a question, which is what we aim to do as well, we continue to use the same terminology in this work.


We hypothesize that better algorithms could be implemented by switching to a learning-based approach.

the simple unsupervised algorithm proposed requires no data annotations, and it captures the crucial intuition that some words in the question contribute more towards the overall information need than others.


Once again, we observe that the performance of the combined model (CR + TAG + embeddings, line 4) is better than the performance of the CR model by itself (line 1) or CR + embeddings (line 3).

here this difference is not significant.


This coincides with pipeline (a) of Figure 11, that is, the induction of a LCFRS of arbitrary fanout.

if the recursive partitioning is transformed by Algorithm 4 or if the left-branching or right-branching recursive partitioning is used (as in Example 20 and as in pipelines (b) and (c) of Figure 11), then the fanout of the induced LCFRS decreases and its derivations are binarized.


This is because, by definition, no cycles occur in the given dependency structure.

if we replace nonterminals of the form J by any other choice of symbols, and combine rules induced by different hybrid trees from a corpus, then cycles may arise if the relationship between inherited and synthesized arguments is confused.


This makes the tree language less refined, and one may expect the scores therefore to be generally lower.

in many cases where strict labeling suffers from a higher proportion of parse failures (so that the parser must fall back on the default structure), it is child labeling that has the higher scores.


With the combination of POS tags and DEPRELs, the tree language can be most accurately described, and we see that this achieves some of the highest UAS and LAS in the case of child labeling.

the scores can be low in the presence of many parse failures, due to the fall-back on the default structure.


With left-branching and right-branching recursive partitionings, parsing tends to be faster than with the other recursive partitionings.

in many cases we do not observe the predicted asymptotic differences, which may be due to larger grammar sizes.


In other words, strictly speaking, +{ and story have different lexical meanings.

contextual information makes up for what is missing in the literal meaning of story (the context for Example (3) is a news story about a thief who left a note on the wall of a home he broke into to encourage the home owner to work harder), and we believe that stories is an appropriate translation for +{ in that context.


This observation is consistent with the findings of other researchers-in particular, those of Lavie, Parlikar, and Ambati (2008) and Ambati and Lavie (2008).

when there is sufficient structure in the parse trees on both sides that are hierarchically aligned, Hiero-style translation rules that capture the translation divergences between Chinese and English can be extracted.


In each case, rejection of the null hypothesis implies dependence between the geographical and linguistic signals.

each test can incorrectly fail to reject the null hypothesis if its assumptions are violated, even if given an arbitrarily large amount of data.


For some methods, it is possible to calculate a p-value from the test statistic using a closed form estimate of the variance.

for consistency, we use a permutation approach to characterize the null distribution over the test statistic values.


After adjusting the p-values using the FDR procedure, a 500-mile cutoff results in three significant linguistic variables.

6 recall that the approach of selecting parameters by maximizing the number of positive test results tends to produce a large number of type I errors.


2013), classifying argument components into claims and premises (Mochales-Palau and Moens 2011;Rooney, Wang, and Browne 2012;Stab and Gurevych 2014b), and identifying argumentative relations (Mochales-Palau and Moens 2009;Peldszus 2014;Stab and Gurevych 2014b).

an approach that covers all subtasks is still missing.


Recently, Peldszus and Stede (2015) proposed an approach based on Minimum Spanning Trees, which jointly models argumentation structures.

it links all argument components in a single tree structure.


Recently, they translated the corpus to English, resulting in the first parallel corpus for computational argumentation.

the corpus does not include non-argumentative text units.


Like convergent arguments, a linked argument includes two premises.

neither of the two premises independently supports the claim.


Some people argue for and others against and there is still no agreement whether cloning technology should be permitted.

as far as I'm concerned, [cloning is an important technology for humankind] MajorClaim1 since [it would be very useful for developing novel cures] Claim1 .


We also experimented with features capturing PDTB relations between the target and source component.

those were not effective for capturing argumentative relations.


Omitting probability and embedding features yields the best accuracy.

we select the best system by means of the macro F1 score, which is more appropriate for imbalanced data sets.


A false negative occurs when an annotator fails to put a unit where she should, namely, where the reference (if any) tells us there should be a unit, and a false positive is the opposite situation.

no reference exists in the case of agreement measures, and there is a symmetry between false positives and false negatives: If annotator 1 puts a unit where annotator 2 doesn't, it is a false positive if we consider annotator 2 as the reference, or a false negative if we consider annotator 1 as the reference.


This leads to an observed categorial agreement of 50% if we consider, as measures do with predefined units, that all units are of the same importance.

if we rely on unit lengths, the first unit counts four times as much as the second one (if we work at word level), and the observed agreement would artificially reach 80% (instead of 50%).


They are inherently and frequently present in unitizing: Because annotators have to put units by themselves on a continuum, it is part of the game that they do not put units where others do.

this question goes beyond the scope of unitizing, and the results of this section concern any categorization measure.


A possible method, which I will call atomization of the continuum, is to compare each atom of the continuum (for instance, at word level) from one annotator to the corresponding atom from another annotator.

this deeply changes the nature of the data (the contiguity of units), and has severe limitations, as demonstrated in Section 3.4.1 of Mathet, Widlöcher, and Métivier (2015).


Figure 12 shows that γ cat and cu α are about the same, decreasing quite regularly from 1 to 0, as expected.

γ goes from 1 to 0.35, because positions are still correct.


Spearman's rank correlation ranges between −1 and 1.

unlike Kendall's τ, here the sign does not matter, and high absolute values indicate better performance.


This tendency is observed across the data sets, and it is quantitatively verified in previous Section 5.2 (i.e., good translations tend to share the tree structure and labels with the reference translations).

translation 12(c) is much more ungrammatical.


The earliest work on using discourse in machine translation that we are aware of dates back to 2000: Marcu, Carlson, and Watanabe (2000) proposed rewriting discourse trees for MT.

this research direction was largely ignored by the research community as the idea was well ahead of its time: Note that it came even before the current standard phrase-based SMT model was envisaged (Koehn, Och, and Marcu 2003).


One way the system can consider the word used by the writer at evaluation time is by proposing a correction only when the confidence of the classifier is high enough.

in this partial solution, the source word is not used in training if the classifier is trained on native data.


Only for verb agreement mistakes, there is about 1 point difference in F1 performance, whereas for the other errors results are quite close.

adaptation using target data substantially outperforms adaptation using unrelated language data, by 1.1 to 2.7 F1 points for different errors.


The F-measure by itself does not indicate what the quality of the text is as a result of running the system; instead, this is done by looking at accuracy and comparing it with the learner baseline.

accuracy measure is only a single point and in order to get improvement it pushes the system toward low recall.


Moreover, adding information about either the position or the dependency labels increases the explained variance for all models.

for the TEXTUAL and LM models the position of the word adds considerable amount of information.


For the VISUAL model it is less clear what to expect: On the one hand, because of their chain structure, RNNs are better at keeping track of short-distance rather than long-distance dependencies and thus we can expect tokens in positions closer to the end of the sentence to be more important.

in English the information structure of a single sentence is expressed via linear ordering: The TOPIC of a sentence appears sentence-initially, and the COMMENT follows.


of the models are indeed different, and that the features encoded by TEXTUAL are more associated with syntactic constructions than in the case of VISUAL.

when comparing LM with TEXTUAL, the difference between context types is much less pronounced, with distributions overlapping.


Using the IMAGINET model as our case study, our analyses of the hidden activation patterns show that the VISUAL model learns an abstract representation of the information structure of a single sentence in the language, and pays selective attention to lexical categories and grammatical functions that carry semantic information.

the language model TEXTUAL is sensitive to features of a more syntactic nature.


This includes tree-structured RNN models such as the Tree-LSTM introduced in Tai, Socher, and Manning (2015), or the CNN architecture of Kim (2014) for sentence classification.

the presented analysis and results regarding word positions can only be meaningful for RNNs as they compute their representations sequentially and are not limited by fixed window sizes.


The classical definition of ungraded lexical entailment is as follows: Given a concept word pair (X, Y), Y is a hypernym of X if and only if X is a type of Y, or equivalently every X is a Y.

4 graded lexical entailment defines the strength of the lexical entailment relation between the two concepts.


The usefulness of these evaluation sets is evident from their wide usage in the LE literature over recent years: They guided the development of semantic research focused on taxonomic relations.

none of the evaluation sets contains graded LE ratings.


Because the main focus of this work is not on the distinction between abstract and concrete concepts, we have not explicitly controlled for the balanced amount of concrete/abstract pairs in HyperLex.

because the source USF data set provides concreteness scores, we believe that HyperLex will also enable various additional analyses regarding this dimension in future work.


2015), which is able to better reason over pairs consisting of two unseen concepts during inference.

shwartz, Goldberg, and Dagan (2016) argue that a random split emulates a more typical "real-life" reasoning scenario, where inference involves an unseen concept pair (X, Y), in which X and/or Y have already been observed separately.


Positive correlation scores for all models reveal that pairs with high graded LE scores naturally imply some degree of semantic similarity (e.g., author / creator).

the scores with similarity-specialized models are much lower than the human performance in the graded LE task, which suggests that they cannot capture intricacies of the task accurately.


IAA scores on both POS subsets are very similar and reasonably high, implying that human raters did not find it more difficult to rate verb pairs.

we observe differences in performance over the two POS-based HyperLex subsets.


The impact of lexical memorization is illustrated by Table 16, using a sample of concept pairs containing "prototypical hypernyms" (Roller and Erk 2016) such as animal: The regression models assign high scores even to clear negatives such as (plant, animal).

the effect of lexical memorization also partially explains the improved performance of all regression models over UNSUPERVISED baselines for a random split, as many (X t , animal) pairs are indeed assigned high scores in the test set.


In current binary evaluation protocols targeting ungraded LE detection and directionality, even simple methods modeling lexical generality are able to yield very accurate predictions.

our preliminary analysis in Section 7.2 demonstrates their fundamental limitations for graded lexical entailment.


they pick their grain of something with the others), meaning that they are benefiting from something as a side effect.

syntactic analysis can help us identify the parts of this MWE that allow for variation, here the determiner that can be changed into a possessive pronoun.


On the one hand, we find MWE identification before translation methods (the so-called static approaches) that concatenate MWEs as a preprocessing step or conversely split compositional closed compounds in Germanic languages to distinguish them from noncompositional compounds.

we find MWE identification during the translation process itself (so-called dynamic approaches).


Evaluating MWE processing extrinsically through the use cases of parsing and MT is an attractive alternative.

as Sections 4 and 5 will further specify, caution is needed when measuring the impact of MWE identification on, for example, parsing quality.


Our survey focuses on interactions of MWE processing with parsing and MT.

we cannot discuss these interactions without providing an overview of approaches in discovery and identification.


In other words, discovery helps extending and creating lexicons that are used in identification.

there are also some challenges in MWE discovery, the most relevant of which are discontiguity and variability.


Discontiguity is generally dealt with when parsing the input corpus prior to MWE discovery (Section 4.1).

syntactic analysis may introduce ambiguities because some MWEs do not allow arbitrary modification and will be wrongly merged with literal uses or accidental cooccurrence (to take turns does not mean that one deviates from a straight trajectory several times.


Word and MWE senses can be modeled using entries of semantic lexicons like WordNet synsets (McCarthy, Venkatapathy, and Joshi 2007).

most discovery methods use distributional models (or word embeddings) instead, where senses are represented as vectors of co-occurring context words (Baldwin et al.


On the one hand, evaluation sets are not always available and, when they are, results may be hard to generalize because of their small size.

large-scale discovery, although potentially useful, is hard to evaluate.


The organization of this chapter is excellent.

i feel that the inclusion of distinct-category SCFG decoding does not fit well into this chapter, as string decoding in string-to-tree SMT requires no knowledge of the source syntax.


The first aspect is the number of teams that do not publish their system's description, which amounts to approximately 9%, and could indeed be related to withdrawals.

exactly because the paper is missing, information on why a team withdrew is not available.


Ranking of systems forms a large part of the appeal of shared tasks.

rankings should not be overemphasized and are far from being the final goal of shared tasks.


The normalization for semirings of this type is O(|Q| 3 ) according to Mohri (2009).

given the restricted graphs that we define, this complete semiring behaves as a k-closed semiring (see our Equations (12) and (13)) because the characteristic matrix M is an upper triangular matrix, and, therefore, the time complexity is O(|δ|).


Given a continuous left-to-right HMM H, a similar algorithm for computing α H (•, •) can be defined.

first, it is necessary to compute the probability of all real sequences that can be generated in a loop.


The size of this chart is O(n2 n ) in the number of ETs-still exponential in the worst case.

as with the more general features, the ENUM algorithm, memoizing recursive calls with the chart, can take advantage of the hard constraints of the UG to lower the time complexity in practice.


Our polynomial-time SAT algorithm does not translate directly into a Viterbi algorithm that is polynomial in the size of the input UG.

we emphasize that for any NP-hard problem with hard constraints, the weighted generalization is also NP-hard.


A tree is a special kind of graph where the vertices are arranged in a hierarchical way, with the property that the set of vertices in any subtree have only one interconnection with the set of the remaining vertices.

for a general graph this is not possible, meaning that we cannot group vertices in a hierarchical structure and pretend that there are a small number of interconnections between vertices in any subtree and the remaining vertices.


This structure does not introduce high treewidth because we can put "and" and each c i into a separate bag, forming a chain of bags of width 1.

when we use the string order as a constraint, we first introduce vertices c 1 , c 2 , .


Furthermore, all of these proposals still retain the stack and buffer architecture of the transition-based dependency parsing system they extend.

the proposal in this article introduces the novel idea of using a cache component in stack-based transition systems.


There are also major notational differences with respect to our proposal: Quernheim and Knight (2012) essentially view computations as top-down rewriting processes, and the rewriting relation is defined via the introduction of specialized DAGs, called incomplete DAGs.

in our definition of run in Sections 3.1 and 7.2, there is no commitment to a specific rewriting process, which makes the notation somewhat simpler.


Similar to our basic (non-extended) model, graph acceptors of this type recognize graph languages of bounded degree.

because of the overlapping of tiles in runs and the occurrence constraint, they are considerably more powerful than our DAG automata (and thus too powerful for our purposes) unless the tiles are required to have the radius 0 (i.e., they are single nodes).


Thus, in this respect our DAG automata are similar to those by Kamimura and Slutzki (1981), whose path languages are trivially regular.

if we restrict recognizable DAG languages to DAGs with only one root, then this no longer holds.


More precisely, if v i (i = 1, 2) is the node such that e i ∈ in(v i ), then D[e 1 ↔ e 2 ] has e 1 ∈ in(v 2 ) and e 2 ∈ in(v 1 ) but is otherwise identical to D. It is not difficult to see that the edge interchange operator we have just defined might introduce cycles, that is, D[e 1 ↔ e 2 ] may no longer be a DAG.

in what follows we will use this operator in a restricted way, such that the resulting graph is still a DAG.


We have thus shown that the path language of every recognizable DAG language is a regular string language.

the proof of this statement relies crucially on the fact that DAGs in [[M]] may have several roots: We considered the disconnected graph the latter may be connected and may thus, in fact, be an element of [[M ]], while containing the roots of both D 1 and D 2 .


For this reason, in the following discussion we use as a term of comparison quantity tw(D), the treewidth of D. When processing D , the running time of Algorithm 2 depends on tw(LG(D )), the treewidth of the line graph of D , which in turn depends on the choice of the binarization of D. There are several ways in which we can binarize D, resulting in different values of tw(LG(D )).

a bad choice of binarization for D may result in tw(LG(D )) much larger than tw(D).


With this guideline, the structure of the book appears smoother from a neural network entry: It first lays the background of neural network methods, and then discusses the traits of natural language data, including challenges to address and sources of information that we can exploit, so that specialized neural network models introduced later are designed in ways that accommodate natural language data.

some fundamentals in natural language processing are not covered in the book, for example, linguistic theories and backgrounds of the natural language processing tasks, and proper preparation of corpus data.