Another approach for automatic labeling is cotraining (Blum and Mitchell, 1998).

with two classifier C 1 and C 2 classifying the same input sentence S with labels l 1 and l 2 , S is labeled as l only if l 1 = l 2 = l. a sentence is labeled and used to train a model only when the two classifiers agree.


Lower perplexity means higher probability of generating the sentence from the dialect model.

sentence S is assigned with label l and used in model re-training if and only if perp l (S) < perp k (S) × threshold, for k = l. The threshold is selected to optimize the classification accuracy on a tuning set.


Compared to other corpora mentioned in the related work, this corpus is the largest one to date that covers different domains and spans several registers of usergenerated Web content.

the corpus comprises four registers (comments to articles, forum posts, blogs, and argumentative newswire articles) and covers six domains related to educational controversies (homeschooling, private vs. public schools, mainstreaming, single-sex education, prayer in schools, and redshirting).


Furthermore, in order to help detecting off-topic and non-argument sentences, we employ topic model features.

we use features taken from a vector representation of the sentence obtained by using Gibbs sampling on LDA model (Blei et al., 2003;McCallum, 2002) with topics trained on unlabeled data provided as a part of the corpus.


We first assess potential biases in sampling techniques for obtaining geotagged Twitter data.

we compare two possible techniques for obtaining data: the location field in the user profile (Poblete et al., 2011;, and the GPS coordinates attached to each message (Cheng et al., 2010;.


Our idea is to exploit the structure of repost tree together with content of messages to help distinguish two different messages on repost tree, i.e., leaders and followers.

leader is referred to as a message on repost tree covering salient new information, which can lead further comments or discussions in its descendant reposts; follower is referred to as a message that contains no comment, simply repeats or naively responds to its ancestor leader message, thus providing no important information.


Different from Basic-LeadSum, every message on repost tree T , no matter detected as a leader or a follower, participates in ranking process of Soft-LeadSum.

in the random walk, visitor wanders on a complete graph G = (V, E ) whose vertex set V is identical to repost tree T , and E = {(u, v)|u ∈ V, v ∈ V, and u = v} represents the edge set.


We utilized ROUGE-N metric (Lin, 2004) for benchmark, which is a standard for evaluating automatic summaries based on N-gram overlapping between a generated summary and a reference.

rOUGE-1 and rOUGE-2 F1-measure were used as our evaluation metrics.


(2013) ranks messages utilizing Gradient Boosted Decision Tree (GBDT) algorithm with text, popularity, temporal and user influence signals to summarize Twitter context tree.

without the interaction data with external users, we utilize users' fol-   GBDT implementation is based on RankLib 5 , and as a supervised method, UserInfSum is evaluated with 10-fold cross validation.


The results demonstrate that all of the proposed classifiers solved the problems with high precision.

for each type, the classifier using a polynomial kernel achieved more than 5 http://www.tkl.iis.u-tokyo.ac.jp/˜ynaga/jdepp/ 70% precision.


The results also show that the classifier using a polynomial kernel for each type outperformed the baseline method based on the state-of-the-art zero anaphora resolution method.

the direct subject sharing recognition using our classifiers has the potential to lead to a significant improvement in zero anaphora resolution, which we confirm through the experiments in Section 7.


This paper presents a detailed comparative framework for assessing the usefulness of unsupervised word representations for identifying so-called implicit discourse relations.

we compare standard one-hot word pair representations against low-dimensional ones based on Brown clusters and word embeddings.


This work also leaves other important questions open.

it is unclear whether all word pairs constructed over the two discourse segments are truly informative and should be included in the model.


This paper fills these gaps and significantly extends the work of (Rutherford and Xue, 2014) by explicitly comparing various types of word representations and vector composition methods.

we investigate three well-known word embeddings, namely Collobert and Weston (Collobert and Weston, 2008), hierarchical log-bilinear model (Mnih and Hinton, 2007) and Hellinger Principal Component Analysis (Lebret and Collobert, 2014), in addition to Brown clusterbased and standard one-hot representations.


, w 2m } the m words in the second segment.


One can thus use mapping Φ all,1,⊕ in lieu of Φ all,1,⊕ : It should be clear that Φ all,1,⊕ provides a normalized version of Φ all,1,⊕ as this latter mapping amounts to weighted versions of the former: Through the comparative framework described in section 3, our objective is to assess the usefulness of different vectorial representations for pairs of discourse segments.

we want to establish whether dense representations are better than sparse ones, and whether certain word pairs are more relevant than others, which resource and which combination schemes are more adapted to the task, and, finally, whether standard features derived from external databases are still relevant in the presence of dense representations.


We use the same classification algorithm for comparing all the described feature configurations.

we train a Maximum Entropy (ME) classifier (aka, logistic regression).


Our work also relates to studies comparing unsupervised representations for other NLP tasks such as name entity recognition, chunking (Turian et al., 2010), sentiment analysis (Lebret and Col-lobert, 2014) or POS tagging (Stratos and Collins, 2005).

we found some similarities between our conclusions and those in (Turian et al., 2010).


In this paper, we approach this problem from the perspective of domain adaptation.

we argue that the reason that automatically-labeled examples generalize poorly is due to domain mismatch from the explicit relations (source domain) to the implicit relations (target domain).


way to obtain performance gains in domain adaptation (Joshi et al., 2012).

our goal is to modify the relation distribution in the source domain (explicitly-marked relations) and make it as similar as possible to the target domain (implicit relations).


For this purpose, we examined the correlation between an error type and its degree of breakdown: the higher the correlation, the more it is related to dialogue breakdown.

we calculated the correlation ratio (η) between the existence of a comment belonging to a particular cluster (error type) and the number of breakdown labels (B labels).


Let U be the set of binary indicator variables corresponding to the values assigned to the back pointers.

u ij = 1 iff a i = j and C is the set of K binary constraint indicator variables indicating if linguistic constraints are violated.


Specifically, u ij = 1 iff a i = j and C is the set of K binary constraint indicator variables indicating if linguistic constraints are violated.

c k,i,j = 1 iff the linguistic constraint C k is violated for the back pointer u ij .


For each edge e k = (p k , i k , o k , w k , n k ) of T , we create an edge ; the 1 is in the k th position.

f k,0 is the tropical weight of the k th edge in T and f k,k = 1 indicates that this tropical weight belongs to edge k in T .


For unsegmented languages, morphological analysis and language modeling form a chicken-and-egg problem.

if high-quality morphological analysis is available, we can learn a high-quality language model from a morphologically analyzed large corpus.


Since the possible context (paths in the word lattice) considered in RNNLM falls into combinatorial explosion in morphological analysis, we keep only probable context candidates inside the beam.

each node keeps candidates inside the beam width.


Our proposed model ("Base + RNNLM retrain ") significantly outperforms all the baseline models and "Base + RNNLM," which does not use retraining.

we achieved a large improvement for segmentation.


We can expect that the word candidates that are given high scores by the perceptron in the symbol grounding result have strong relationship to the positions.

we can make a good dictionary by selecting word candidates in descending order of the scores.


In the rest of this section, we explain model variants in detail.

subsections 3.1 and 3.2 address the first and second problems; subsection 3.3 deals with the third problem by introducing several pooling heuristics.


Figure 1 shows the structure of a LSTM unit.

these gates and the memory cell allow a LSTM unit to adaptively forget, memorize and expose the memory content.


the training data is not fully representative of the test data.


Instead, it represents a set of similarity values between document d and the center of the positive documents.

the learning consists of the following steps:  where Sim is a similarity function consisting of a set of similarity measures.


Instead, we still perform the traditional topic based classification.

given a review, the system decides what type of product the review is about.


In our model, we use the content of a paper as one view and the citation contexts as another view of our data.

for the content of a paper, we use its title and abstract as it is commonly used in the literature (Lu and Getoor, 2003); for the citation contexts, we use both the cited and citing contexts, as described in the previous section.


Besides humor theories and linguistic style modeling, one important theory behind humor is its social/hostility focus, especially regarding its interpersonal effect on receivers.

humor is essentially associated with sentiment (Zhang and Liu, 2014) and subjectivity (Wiebe and Mihalcea, 2006).


Our data is drawn from Part 1 of the Fisher English speech corpus (Cieri et al., 2004c), which contains audio recordings (Cieri et al., 2004a) and manual transcriptions (Cieri et al., 2004b) of telephone conversations.

we use the topic ID training and evaluation test subsets defined in prior work (Hazen et al., 2007).


draws from a document-level multinomial φ (d) ; as in LSA, each latent component contributes equally to each word within a given document.

in the mi-vector model, the natural parameter vector of the multinomial for all words in a given document is determined by an additive offset from a background parameter vector.


On our subset of the Fisher English data, these tasks appear to have competing objectives: the best representations in one task are not necessarily the best in the other.

while SAGE yields the worst performance as a feature learner for topic ID, it is demonstrably superior to other low-dimensional learned representations in topic discovery.


This choice of experimental protocol is a nuisance variable in our results; other partition constructions may yield different conclusions.

the heaviestcoordinate partition may favor topic models, whose representations are probability vectors, and disfavor mi-vectors and LSA, whose representations may have positive and negative coordinates encoding general linear combinations.


Future work could evaluate other implementations of LSA or use a higher-level view of speech, such as triphone state cluster ngrams, that more naturally exhibits sparsity and lends to tf-idf weighting.

weighting by a likelihood ratio test statistic and applying a log transform has generated better performance in several other tasks (Lapesa and Evert, 2014).


m =< e 1 , e 2 , r, k > (1) Similar with most of the OAEI systems (Li et al., 2009; Ngo and Bellahsene, 2012; Cheatham and Hitzler, 2013b), we focus on discovering only equivalent mappings between classes and properties with cardinality 1:1.

one class (property) in ontology O 1 can be matched to at most one class (property) in ontology O 2 and vise versa.


We first generate a combination of the entity's descriptions (CDs for short) and then create a term vector for each entity.

the CDs of a class = the class's ID + label + comments + its properties' descriptions + its instances' descriptions.


In this way, when we input the binary term vector to the network, we can get its abstract representations in different levels.

with the layer-by-layer learning, we obtain different levels of representations.


For Relative Neighborhood Graphs (RNG) this region between two points a and c is defined as the intersection of the two spheres with centers in a and c with radius d(a, c).

a point b lies between points a and c if it is closer to both a and c than a and c are to each other.


For the linguistic representations we use the continuous vector representations from the log-linear skip-gram model of .

we trained 300-dimensional vector representations trained on a dump of the English Wikipedia plus newswire (8 billion words in total).


The objective for the multi-modal skip-gram has an additional visual objective J vis (in this case a margin criterion): Here, we take a similar but more straightforward approach by making the auditory context a part of the initial training objective, which is possible because linguistic and auditory representations have the same dimensionality.

we modified word2vec to predict additional auditory contexts that have been set to the corresponding BoAW representation.


They do not attempt to categorize arbitrary sentences in 'free text', however, but work with a corpus of selected sentences and use gold standard parse information for their experiments.

they consider clauses containing lexically dynamic verbs only.


The input to the model are instances of arguments (e.g., window, the burglar in sentence (1)) and the output is a grouping of these instances into clusters such that each cluster contains arguments corresponding to a specific semantic role and each role corresponds to exactly one cluster.

the syntactic representations of verbal predicates, and argument positions are observable, whereas the associated semantic roles are latent and need to be inferred.


In this work we propose to learn these features and their complex interactions (e.g., selectional restrictions) automatically from data.

we induce embeddings to represent a predicate and its arguments.


We compared our model against a baseline that assigns arguments to clusters based on their syntactic functions (SYNTF; Lang and Lapata, 2014).

the baseline forms clusters from the syntactic position of an argument using four cues: the verb's voice, the argument's position relative to the predicate, its syntactic relation, and any realizing preposition.


Existing work treats both labeling sources as ontologically interchangeable.

it does not matter whether a text was labeled by the author in the process of writing said text, or by an annotator who has been paid to label the text a posteriori.


We propose a novel data augmentation approach to enhance computational behavioral analysis using social media text.

we collect a Twitter corpus of the descriptions of annoying behaviors using the #petpeeve hashtags.


In this section, we describe our methods for the qualitative and quantitative analyses.

we briefly review a supervised approach of using sparse mixed-effects topic model to visualize the topical words to analyze this behavior data.


Batches of 100 are the maximum for M-CT and T-Lex, but we go up to 1000 with A-CT. As expected, the performance of the classifiers trained on individual tweets is poor (see Figure 1).

the models trained on A-CT and M-CT data have very low accuracy on data with 1 bit payload, performing only slightly better than random guessing.


To collect texts from which the text patterns between entities and numerical values will be extracted we downloaded documents from the web.

for each region combined with each property we formed a query consisting of the two and submitted it to Bing via its Search API.


We also report results on the French treebank, with one small change to network structure.

we also included morphological features for each word as input to the recurrent network, using a small embedding for each such feature, to demonstrate that our parsing model is able to exploit such additional features.


In our scheme, a back-off state (or function) q b produces estimates that are target structures t 2 ∈ T ∆ with score s ∈ R, given some information of the source tree fragment t 1 ∈ T Σ .

a function q b : In our QA application, we only use the leaves of the input subtree t 1 and use lexicons or entity/predicate linkers to retrieve KB entities, KB relations or a compound of a disambiguating relation and an entity from backoff states ent, pred and bridge, respectively.


In this work we rely on a version of the PTB released by Ficler and Goldberg (2016) in which the above deficiencies are manually resolved.

coordinating elements, coordination phrases and conjunct boundaries are explicitly marked with specialized function labels.


We approach this by training the similarity function in a data-dependent manner.

we train an encoder that encodes spans into vectors such that vectors of similar spans will have a small Euclidean distance between them.


For example, consider the sentence in In order to model the validity of the connection points, we represent each connection point as the concatenation of a forward and reverse LSTMs centered around that point.

for the spans in Figure 3 the two connection points are represented as: Formally, assuming words w 1:n in a sentence with coordination at position k and conjuncts w i:j and w l:m , 5 the connection points are between w 1:j and w m+1:n ; and between w 1:i−1 and w l:n .


Candidates Extraction We extract candidate spans based on the inside-outside probabilities assigned by the Berkeley parser.

to obtain candidates for conjunct span we collect spans that are marked with COORD, are adjacent to the coordinating word, and have non-zero inside or outside probabilities.


Our main empirical finding is that though two biases, avoiding center-embedding and favoring shorter dependencies, are conceptually similar (both favor simpler grammars), often they capture different aspects of syntax, leading to different grammars.

our bias cooperates well with additional small syntactic cue such as the one that the sentence root tends to be a verb or a noun, with which our models compete with the strong baseline relying on a larger number of hand crafted rules on POS tags (Naseem et al., 2010).


Stack depth and center-embedding We use the term center-embedding to distinguish just the tree structures, i.e., ignoring symbols.

the tree in Figure 4(a) is the minimal, one degree of centerembedded tree, where the constituent rooted at A is embedded into a larger constituent rooted at D. Multiple, or degree ≥ 2 of center-embedding occurs if this constituent is also embedded into another larger constituent.


Besides these simple ones, we also investigate relaxing δ = 1 that results in an intermediate between δ = 1 and 2.

we relax the depth calculation in COMP (Eq.


4 Spurious ambiguity The central issue for applying LC parsing is the spurious ambiguity in dependency grammars.

there are more than one (binary) CFG parses corresponding to a given dependency tree.


This observation indicates that our restricted grammar implicitly binarizes a dependency tree, and the incurred stack depth (or the degree of center-embedding) is determined based on the structure of the binarized tree.

we can show that the presented grammar performs optimal binarization; i.e., it minimizes the incurred stack depth.


8 Constraining root word We also see the effects of the constraints when a small amount of grammatical rule is provided.

we restrict the candidate root words of the sentence to a noun or a verb; similar rules have been encoded in past work such as Gimpel and Smith (2012) and the CCG induction system of Bisk and Hockenmaier (2013).


We feel the results are expected observing that center-embedded constructions are a special case of longer dependency constructions.

lEN is a stronger constraint than DEP in that the structures penalized by DEP are only a subset of structures penalized by lEN.


We have shown that a syntactic constraint that eliminates center-embedding is helpful in dependency grammar induction.

we found that our method facilitates to find linguistically correct constituent structures, and given an additional cue on dependency, the models compete with the sys-tem relying on a significant amount of prior linguistic knowledge.


Future work includes applying our DEP constraint into other PCFG-based grammar induction tasks beyond dependency grammars.

it would be fruitful to apply our idea into constituent structure induction for which, to our knowledge, there has been no successful PCFGbased learning algorithm.


Syntactic analyses produced by syntactic parsers are a long way from representing the full meaning of the sentences parsed.

they cannot support questions like "Who did what to whom?


Furthermore, these propositions are not complete.

queries of the form 'What is the other name of Rozsa Hill?


Since it is not possible to enumerate all the different syntactic variations of an assertion, there is a need to learn general patterns that encode the various ways of expressing the assertion.

we learn the various syntactic patterns that can encode the same information as the seed patterns and hence can be mapped to same representation.


(2013), which uses 7 tags to handle entities that can be discontiguous and overlapping.

we used B, I, O, BD, ID, BH, and IH to denote Beginning of entity, Inside entity, Outside of entity, Beginning of Discontiguous entity, Inside of Discontiguous entity, Beginning of Head, and Inside of Head.


A hypergraph can be used to represent entities of different types and their combinations in a given sentence.


Can the Tradeoff Hypothesis be implemented in an unsupervised model that predicts skipping and reading times in quantitative terms?

can we compute surprisal based only on the words that are actually fixated?


While some differences between simulated and human fixation probabilities can be noticed, there are similarities in the general qualitative features of the two heatmaps.

function words and short words are less likely to be fixated than content words and longer words in both the simulated and the human data.


We therefore derive model predictions by rescaling the simulated fixation probabilities so that their average equals the fixation rate in the development set, and then greedily take the maximum-likelihood sequence.

we predict a fixation if the rescaled probability is greater than 0.5, and a skip otherwise.


Importantly, in contrast to the AB model, the verb bias score of the BFS model also significantly correlates with the judgment data.

the BFS model provides a better prediction of the preference per verb, which is key to producing a verb in the appropriate syntax.


We therefore assume that our model with rationales is trained on the same data as the original neural models, without access to additional rationale annotations.

target rationales are never provided during training; the intermediate step of rationale generation is guided only by the two desiderata discussed above.


In order for the subset to qualify as a rationale it should satisfy two criteria: 1) the selected words should be interpretable and 2) they ought to suffice to reach nearly the same prediction (target vector) as the original input.

a rationale must be short and sufficient.


The target vector is then generated on the basis of the final state reached by the recurrent unit after processing all the words in the input sequence.

generator gen(•): The rationale generator extracts a subset of text from the original input x to function as an interpretable summary.


In a simple generator, the probability that the t th word is selected can be assumed to be conditionally independent from other selections given the input x.

the joint probability p(z|x) factors according to The component distributions p(z t |x) can be modeled using a shared bi-directional recurrent neural network.


That is, the joint probability p(z|x) factors according to The component distributions p(z t |x) can be modeled using a shared bi-directional recurrent neural network.

let − → f () and ← − f () be the forward and backward recurrent unit, respectively, then Independent but context dependent selection of words is often sufficient.


First, the rationale that it produces must suffice as a replacement for the input text.

the target vector (sentiment) arising from the rationale should be close to the gold sentiment.


Before training the joint model, it is worth assessing the neural encoder separately to check how accurately the neural network predicts the sentiment.

to this end, we compare neural encoders with bigram SVM model, training medium and large SVM models using 260k and all 4 for each aspect we train a simple linear regression model to predict the rating of this aspect given the ratings of the other four aspects.


Using causality as a use case, we answer this question with a framework for producing causal word embeddings with minimal supervision, and a demonstration that such taskspecific embeddings significantly benefit causal QA.

the contributions of this work are: (1) A methodology for generating causal embeddings cost-effectively by bootstrapping cause-effect pairs extracted from free text using a small set of seed patterns, e.g., X causes Y.


Additionally, embeddings have been customized to distinguish functional similarity from relatedness (Levy and Goldberg, 2014;Kiela et al., 2015).

levy and Goldberg train their embeddings by replacing the standard linear context of the target word with context derived from the syntactic dependency graph of the sentence.


(2013), which has been shown to improve a variety of language processing tasks including QA (Yih et al., 2013;Fried et al., 2015).

we use the variant implemented by Levy and Goldberg (2014) which modifies the original algorithm to use an arbitrary, rather than linear, context.


For this, we first score the tuples by their causal PMI and then scale these scores by the overall frequency of the tuple (Riloff, 1996), to account for the PMI bias toward low-frequency items.

the score S of a tuple, t, is computed as: We then discretize these scores into five quantiles, ascribing a linearly decreasing weight during training to datums in lower scoring quantiles.


For this evaluation, we follow the protocol of Levy and Goldberg (2014).

we create a collection of word pairs, half of which are causally related, with the other half consisting of other relations.


Because this sparsity is partially driven by directionality, we implemented a bidirectional embedding model (cEm-bedBi) that (a) trains a second embedding model by reversing the input (effects as targets, causes as contexts), and (b) ranks pairs by the average of the scores returned by these two unidirectional causal embedding models.

the final bidirectional score of the pair, (e 1 , e 2 ), where e 1 is the can-didate cause and e 2 is the candidate effect, is: s bi (e 1 , e 2 ) = 1 2 (s c→e (e 1 , e 2 ) + s e→c (e 2 , e 1 )) (2) where s c→e is the score given by the original causal embeddings, i.e., from cause to effect, and s e→c is the score given by the reversed-input causal embeddings, i.e., from effect to cause.


In this architecture, the candidate answers are initially extracted and ranked using a shallow candidate retrieval (CR) component that uses solely information retrieval techniques, then they are re-ranked using a "learning to rank" approach.

we used SVM rank 8 , a Support Vector Machines classifier adapted for ranking, and re-ranked the candidate answers with a set of features derived from both the initial CR score and the models we have introduced.


However, the information required to execute this logical form and answer the question must be extracted from the diagram.

eATS(x, y) depends on whether an arrow is present between x and y, which we must train a vision model to determine.


Figure 4.2 illustrates this tree for the logical form λf.CAUSE(DECREASE(MICE), f (SNAKES)), which # initialize predicate instance variables # from text labels in environment world = {"mice": undef, ("mice", "snakes"): undef, ..  corresponds to the question "what happens to the snakes when the mice decrease?"

this logical form is shorthand for the following program: filter(lambda f.cause( decrease(getOrganism("mice")), f(getOrganism("snakes"))), set(decrease, increase, unchanged)) entities such as MICE are created by calling getOrganism and logical forms with functional types implicitly represent filters over the appropriate argument type.


A major difference between what the state-of-theart models do and what our model does is that our model distributes a probability mass of one across the vocabulary, while the thematic fit models have no such overall constraint; they will assign a high number to all words that are similar to the prototypical vector, without having to distribute probability mass.

this implies that two synonymous fillers, one of which is a frequent word like fire, and the other of which is an infrequent word, e.g., blaze, will get similar ratings by the distributional similarity models, but quite different ratings by the neural network model, as the more frequent word will have higher probability.


"Accuracy 2" counts a hit iff, given an object, the composed subject-verb combination gives a higher score when the subject is better fitting.

a hit is achieved when journalist check spelling has a higher score than mechanic check spelling, setting the query to the model as journalist check and mechanic check and finding a score for spelling in that context.


the optimal sense sequence of the sentence) that minimize the energy function in Equation 1: We use dynamic programming to do inference which is similar to the Viterbi algorithm of the hidden Markov model.

for every valid assignment as the energy of the best sense sequence up to position i − 1 that is consistent with the assignment A i−2k , .


The key idea is to represent and model triples and rules in a unified framework.

triples are represented as atomic formulae and modeled by the translation assumption, while rules represented as complex formulae and modeled by t-norm fuzzy logics.


Experimental results show that joint embedding brings significant and consistent improvements over stateof-the-art methods.

it enhances the prediction of new facts which cannot even be directly inferred by pure logical inference, demonstrating the capability of our method to learn more predictive embeddings.


Experimental results show significant and consistent improvements over state-of-the-art methods.

joint embedding enhances the prediction of new facts which cannot even be directly inferred by pure logical inference, demonstrating the capability of KALE to learn more predictive embeddings.


To model triples we follow TransE , as it is simple and efficient while achieving state-of-the-art predictive performance.

given a triple (e i , r k , e j ), we model the relation embedding r k as a translation between the entity embeddings e i and e j , i.e., we want e i + r k ≈ e j when the triple holds.


triples are represented as atomic formulae and modeled by the translation assumption, while rules as complex formulae and by the t-norm fuzzy logics.


This approach harnesses large amounts of automatically extracted discourse connectives along with their arguments to construct new distributional word representations.

we represent words in the space of discourse connectives as a way to directly encode their rhetorical function.


Before turning to the details of how we construct our distributional connective-based model, note that we decided to learn a unique representation for any individual word, irrespective of its position (with)in a particular segment.

we represent both arguments of a connective as a single bag of words.


We therefore rely on automatic annotation of raw data, instead of using the relatively small amount of explicit examples manually annotated in the PDTB (roughly 18, 000 examples).

we used the Bllip corpus 3 composed of news articles from the LA Times, the Washington Post, the New York Times and Reuters and containing 310 millions of words automatically POS-tagged.


Our experiments investigate the relevance of our connective-based representations for implicit discourse relation identification, recast here as multiclass classification problem.

we aim at evaluating the usefulness of having a word representation linked to the task, compared to using generic  word representations (either one-hot, cluster-based or distributed), and whether they encode all the information relevant to the task, thus comparing systems with or without additional hand-crafted features.


In this paper, we aim to extend the main idea behind SCL to neural network-based solutions to sentiment classification to address the domain adaptation problem.

we borrow the idea of using pivot prediction tasks from SCL.


With the two auxiliary tasks, we can learn a neural network model in a standard way to produce sentence embeddings that work well for the auxiliary tasks.

we still use Θ to denote the parameters of the neural network that produces the sentence embeddings (and f Θ the corresponding transformation function), and we use β + and β − to denote the parameters of two softmax classifiers for the two auxiliary tasks, respectively.


This way, the learning of Θ depends not only on D a but also on D s , i.e., the sentimentlabeled training data from the source domain.

we use Θ to denote the parameters for a neural network that takes the original sentence x and transforms it to a sentence embedding (and f Θ the corresponding transformation function).


In this paper, we adopt a CNN-based approach because it has been shown to work well for sentiment classification.

each word (including the token UNK) is represented by a word embedding vector.


In Table 3, we also show the comparison between mDA-NN and our model under a setting some labeled target data is used.

we randomly select 100 sentences from the development set and mix them with the training set.


Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT).

at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-ofthe-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences.


For this reason, the two following analyses rely on mTER.

we investigate how specific characteristics of input documents affect the system's overall translation quality, focusing on (i) sentence length and (ii) the different talks composing the dataset.


We can see that NMT generates translations which are morphologically more correct than the other systems.

the %∆ for NMT (-13.7) is lower than that of the second best system (PBSY, -16.9) by 3.2% absolute points, leading to a percentage gain of around 19%.


In this paper, we conduct in-depth investigation into the recently proposed multi-way, multilingual neural machine translation .

we are interested in its potential for zero-resource machine translation, in which there does not exist any direct parallel examples between a target language pair.


For instance, in our case, Es→Fr can be decomposed into a sequence of Es→En and En→Fr.

we translate a source sentence (Es) into a pivot language (En) and then translate the English translation into a target language (Fr), all within the same multi-way, multilingual model trained by using bilingual corpora.


This allows for more flexible forms of anchors with weak supervision.

for each state h ∈ H, let its set of anchor words be A(h) is the set of unambiguous words that always take the label h. This can be estimated from the labeled dataset D L by collecting the most frequent unambiguous words for each label.


We argue that the effective strategy is to learn from both explicit and implicit supervision signals jointly.

we design a novel online learning algorithm based on structured-output Perceptron.


Indeed, in Line 10, the algorithm will not perform an update if the model could not find any parses matching the implicit signals in Line 9.

if ∆(E(y), z i ) = 1 for all y ∈ Ω, y =ŷ due to the tie-breaking mechanism.


We set the parameter γ to 0.5 unless otherwise stated.

the first half of the training iterations use only explicit supervision.


We also analyze the impact of the parameter K, which controls the size of the candidate set Ω in MixedSP.

for DRAW-1K, when setting K to 5 and 10, the accuracy of MixedSP is at 59.5%.


paraphrase detection (Ganitkevitch et al., 2013;Barzilay and Lee, 2003;Dolan et al., 2004) and evaluation (Chen and Dolan, 2011), question answering (Fader et al., 2013), textual entailment (Bosma and Callison-Burch, 2007), machine translation (Marton et al., 2009), short answer scoring (Koleva et al., 2014), domain adaptation of dependency parsing (Choe and McClosky, 2015).

in ATS, these techniques are typically applied to already existing parallel corpora; in this case the task of aligning the original sentence to its corresponding simple version can be tackled by applying similarity metrics that consider the TF/IDF score of the words in the sentence (Barzilay and Elhadad, 2003;Nelken and Shieber, 2006;Coster and Kauchak, 2011) or methods taking into account also the order in which information is presented (Bott and Saggion, 2011).


This subset was manually reviewed by two nativespeaker linguists with a background in text simplification.

they reviewed a subset of 10,543 pairs at different cosine similarity scores, i.e.


The relation between readability score and sentence operations is shown in Figure 5.

we calculated how the distribution of operations changes with increasing differences between the readability score assigned to the complex and the simple sentence of each pair.


Although it is difficult to study the effect of each sin- gle operation on the readability score variation since these operations are usually applied in combination, we observe some clear tendencies.

operations concerning deletion and verbal features are the most frequent ones both at lower and higher readability scores differences.


For what concerns the other operations, the trend is quite homogeneous along with the different readability scores.

this is the case of reordering thus showing the proposed approach is able to intercept syntactic transformations which impact at different readability variations.


Considering the last few tokens of a sequence normally have more influence on the representation of the whole sequence learnt with LSTM because they get through less times of forget gate from the LSTM computation unit, to effectively capture the information from both beginning and end of an EDU, we use bidirectional LSTM to learn the representation of an EDU.

one LSTM takes the word sequence in forward order as input, the other takes the word sequence in reversed order as input.


The convolution result of each kernel, termed conv i , is further activated with the RELU non-linear activation function (Xu et al., 2015), which is formulated as: Pooling Layer: Because utterance and response are naturally variable-sized, we put a max-overtime pooling layer on the top of temporal convolutional layer (Kim, 2014), which extracts the max value for each kernel, and gets a fixsized representation of length k for utterance and response.

representations constructed by CNN with max-pooling reflect the core meanings of utterance and response.


Comparing with the state-of-the-art results of previous work (Ji and Eisenstein, 2015;Rutherford and Xue, 2015), VarNDRR achieves comparable results in term of the F1 scores.

varNDRR outperforms Rutherford and Xue (2015) on EXP, and Ji and Eisenstein (2015) on TEM.


However, the change trend of the F1 score does not follow that of the lower bound which takes more time to converge.

to the four discourse relations, we further observe that the change paths of the F1 score are completely different.


This may suggest that the four discourse relations have different properties and distributions.

the number of epochs when the best F1 score reaches is also different for the four discourse relations.


We experiment with different methods to convert event components into vector representations.

we use Explicit Semantic Analysis (ESA), Brown Cluster (BC), Word2Vec (W2V) and Dependency-Based Word Embedding (DEP) respectively to convert text into vectors.


Dependency-Based Embedding DEP is the generalization of the skip-gram model with negative sampling to include arbitrary contexts.

it deals with dependency-based contexts, and produces markedly different embeddings.


Using the set of taxonomic relations extracted from the first step as reference, we extract from the Wikipedia corpus all sentences which contain at least two terms involved in a taxonomic relation.

for each sentence, we use the Stanford parser (Manning et al., 2014) to parse it, and check whether there is any pair of terms which are nouns or noun phrases in the sentence having a taxonomic relationship.


In this paper, instead of explicitly discovering the contextual patterns of taxonomic relations, we propose a dynamic weighting neural network to encode this information, together with the hypernym and hyponym, for learning term embeddings.

the target of the neural network is to predict the hypernym term from the given hyponym term and contextual words.


To decide whether a term x is a hypernym of term y, we build a classifier that uses embedding vectors as features for taxonomic relation identification.

we use Support Vector Machine (SVM) (Cortes and Vapnik, 1995) for this purpose.


Furthermore, to evaluate the effect of the offset vector to taxonomic relation identification, we deploy a setting that removes the offset vector in the feature vectors of SVM.

for SVM+Our and SVM+Word2Vec, the input vector is changed We use the subscript short to denote this setting.


Our approach achieves significantly better performance than Yu's method and Word2Vec method in terms of accuracy (t-test, p-value < 0.05) for both BLESS and ENTAILMENT datasets.

our approach improves the average accuracy by 4% compared to Yu's method, and by 9% compared to the Word2Vec method.


We can observe that not only for general domain datasets but also for specific domain datasets, our term embedding learning approach has achieved significantly better performance than Yu's method and the Word2Vec method in terms of accuracy (ttest, p-value < 0.05).

our approach improves the average accuracy by 22% compared to Yu's method, and by 9% compared to the Word2Vec method.


In this section, we evaluate performance of different methods on the Relation Schema Induction (RSI) task.


From this figure, we find that SICTF is able to achieve a 14x speedup on average over KB-LDA 10 .

sICTF is not only able to Ablation MEDLINE stackOverflow A1 A2 Avg A1 A2 Avg sICTF 0.64 0.64 0.64 0.96 0.92 0.94 sICTF (λ rel = 0) 0.60 0.56 0.58 0.83 0.70 0.77 sICTF (λ np = 0) 0.46 0.40 0.43 0.89 0.90 0.90 sICTF (λ rel =0, λ np = 0) 0.46 0.50 0.48 0.84 0.33 0.59 sICTF (λ rel =0, λ np = 0, and no non-negativity constraints ) 0.14 0.10 0.12 0.20 0.14 0.17 Table 6: RsI accuracy comparison of sICTF with its ablated versions when no relation side information is used (λ rel = 0), when no NP side information is used (λnp = 0), when no side information of any kind is used (λ rel = 0, λnp = 0), and when additionally there are no non-negative constraints.


This expansion step results in much larger sets W * and K * , where |W * | = 18,291,330 and |K * | = 15,362,268.

they are 3 and 11 times bigger than the original training sets described in Section 3.1.


The performance of TAXOEMBED is evaluated by conducting several experiments, both automatic and manual.

we assess its ability to return valid hypernyms for a given unseen term with a held-out evaluation dataset of 250 Wikidata termhypernym pairs (Section 5.1).


As Table 3 shows, the data sources vary in coverage: it would be unfair to compare data sources on the different subsets of the evaluation datasets that they have coverage for.

when comparing data sources we want to make sure we evaluate on images for the exact same word pairs.


The objective of this study has been to systematically compare network architectures and data sources for multi-modal systems.

we focused on the capabilities of deep visual representations in capturing semantics, as measured by correlation with human similarity and relatedness judgments.


Mixed Assignment We next define the concept of a mixed assignment: Definition 1.

for a training pair (x, y) and a predicted label y * ∈ Y(x), y * = y, a mixed assignment (M A) vector denoted as m J (y * , y) is defined with respect to J ∈ JJ x as follows: a mixed assignment is a new label, derived from the predicted label y * , that is identical to y * in all indexes in J and to y otherwise.


This metric measures the alignment between word embeddings and a set of lexical semantic features.

we use the semcor noun verb supersenses oracle provided at the qvec github repository.


This enables us to use standard stochastic gradient optimization for training the proposed model.

there are three essential components in VNMT (The detailed architecture is illustrated in Figure 2): • A variational neural encoder transforms source/target sentence into distributed representations, which is the same as the encoder of NMT ) (see section 3.1).


Intuitively, the reparameterization bridges the gap between the generation model p θ (y|z, x) and the inference model q φ (z|x, y).


We find that the performance curve of our VNMT model always appears to be on top of that of GroundHog with a certain margin.

on the final group with the longest source sentences, our VNMT obtains the biggest improvement (3.55 BLEU points).


However, there are essential differences between GroundHog and our VNMT.

groundHog does not translate the phrase "官 员" at the beginning of the source sentence.


For this model, we derive its convex relaxation and show that it too has strong performance despite not having the local optima problems of non-convex objectives.

the word alignment quality of this new convex model is significantly above that of the standard IBM Models 2 and 3, as well as the popular (and still non-convex) IBM Model 2 variant of (Dyer et al., 2013).


The HMM is the last model whose expectation step is both exact and simple, and it attains a level of accuracy that is very close to the results achieved by much more complex models.

experiments have shown that IBM Models 1, 2, and 3 all perform worse than the HMM and Model 4 benefits greatly from being seeded by the HMM (Och and Ney, 2003).


Let (t, d) be all the parameters of the HMM.

as a first step in getting a convex HMM, one could follow the path developed in (Simion et al., 2015a) and directly replace the HMM's objective terms the geometric mean function is convex ( (Boyd and Vandenberghe, 2004)) and, for a given sentence pair (e (k) , f (k) ) with alignment a (k) we can find a projection matrix P so that j=1 are exactly the parameters used in the term above (in particular, t, d are the set of all parameters whilet,d are the set of parameters for the specific training pair k; P projects from the full space onto only the parameters used for training pair k).


The above analysis motivates why we defined our model as we did: we now have only two terms to relax.

to rectify the above, we left in place the structure discussed in Section 3 and made λ and θ be tuning parameters which we can cross-validate for on a small held-out data set.


Using this setup we now have a convex objective to optimize over.

we've formulated a convex relaxation of the IBM2-HMM problem which, like the Support Vector Machine, includes parameters that can be cross-validated over (Boyd and Vandenberghe, 2004).


Because of this, the most natural method of evaluating the quality of the parameters would be to use the same rule as the HMM.

for a sentence pair (e, f ) with |e| = l and |f | = m, in HMM decoding we aim to find (a 1 .


We test several models in our experiments.

we empirically evaluate our models against the GIZA++ IBM Model 3 and HMM, as well as the FastAlign IBM Model 2 implementation of (Dyer et al., 2013) that uses Variational Bayes.


In training, we employ the standard practice of initializing non-convex alignment models with simpler non-convex models.

we initialize, the GIZA++ HMM with IBM Model 2, IBM Model 2 with IBM Model 1, and IBM2-HMM and IBM Model 3 with IBM Model 2 preceded by Model 1.


Second, we obtain distributed representations of words and relations by leveraging a novel word embedding method that considers the multi-sense nature of words and the relational knowledge among words (or their senses) contained in dictionaries.

for each polysemous word, we retrieve its number of senses from a dictionary, and conduct clustering on all its context windows in the corpus.


Furthermore, in addition to learning the embedding vectors for words, we also learn the embedding vectors for relations (e.g., synonym and antonym) at the same time, by incorporating relational knowledge into the objective function of the word embedding learning algorithm.

the learning of word-sense representations and relation representations interacts with each other, such that the relational knowledge obtained from dictionaries is effectively incorporated.


Since different types of questions have their unique ways of expression, the classification task is relatively easy, and we therefore take a simple approach to fulfill the task.

we regard each verbal question as a short document and use the TF•IDF features to build its representation.


To address this problem, we propose a novel approach that considers the multi-sense nature of words and integrate the relational knowledge among words (or their senses) into the learning process.


We use TF•IDF to define the weighting function, where we regard each context window of the word as a short document to calculate the document frequency.

for a word w 0 , each of its context window can be de- Then we represent the window by calculating the weighted average of the pre-learned embedding vectors of the context words as the TF•IDF score of w i , and v w i is the pre-learned embedding vector of w i .


we removeξ i and ζ j from the cluster vector set and the sense vector set, and recursively run (1) to find the next matched pair till all the pairs are found.


We propose to use a function E r as described below to capture the relational knowledge.

the existing relational knowledge extracted from dictionaries, such as synonym, antonym, etc., can be naturally represented in the form of a triplet (head, relation, tail) (denoted by (h i , r, t j ) ∈ S, where S is the set of relational knowledge), which consists of two word-sense pairs (i.e.


The second solver is based on the minimization objective of the translation distance between entities in the relational knowledge model (2).

we calculate the offset vector between the embedding of question word w q and each word w j in the candidate list.


Similar to solving the Synonym questions, we explored two solvers for Antonym questions as well.

the first solver (7) is based on the small offset distance between semantically close words whereas the second solver (8) leverages the translation distance between two words' offset and the embedding vector of the antonym relation.


Therefore, we only use the online questions to train the verbal question classifier described in Section 3.1.

we manually collected and labeled 30 verbal questions from the online IQ test Websites 3 for each of the five types (i.e.


Latent Dirichlet Allocation (LDA) (Blei et al., 2003).

we trained word representations using LDA on wiki2014 with the topic number 1000.


In this baseline, we applied the word embedding trained by skipgram (Mikolov et al., 2013) (denoted by SG-1) on wiki2014.

we set the window size as 5, the embedding dimension as 500, the negative sampling count as 3, and the epoch number as 3.


This is our proposed method in Section 3.

when learning the embedding on wiki2014, we set the window size as 5, the embedding dimension as 500, the negative sampling count as 3, and the epoch number as 3.


From this table, we observe: (i) RK can achieve the best overall accuracy than all the other methods.

rK can raise the overall accuracy by about 4.63% over HP 7 .


We investigated how to automatically solve verbal comprehension questions in IQ Tests by using word embedding techniques.

we proposed a three-step framework: (i) to recognize the specific type of a verbal comprehension question by a classifier, (ii) to leverage a novel deep learning model to co-learn the representations of both wordsense pairs and relations among words (or their senses), (iii) to design dedicated solvers, based on the obtained word-sense pair representations and relation representations, for addressing each type of questions.


We use a similar approach to tackle this task with LSTMNs.

we use two LSTMNs to read the premise and hypothesis, and then match them by comparing their hidden state tapes.


We compared variants of our model against different types of LSTMs (see the second block in Table 3).

these include a model which encodes the premise and hypothesis independently with two LSTMs (Bowman et al., 2015), a shared LSTM (Rocktäschel et al., 2016), a word-by-word attention model (Rocktäschel et al., 2016), and a matching LSTM (mLSTM; Wang and Jiang (2016)).


Suppose E is a set of entities, L a set of literals (I = E ∪ L is also called individuals), C a set of classes, and R a set of directed relations, a knowledge base K consists of two parts: an ontology O ⊆ C × R × C and a model M ⊆ E × R × (C ∪ E ∪ L).

an ontology specifies classes and relations between classes, and a model consists of facts about individuals.


The dataset presents a high diversity and covers a wide range of domains including People, Astronomy, Medicine, etc.

it contains 148, 506, 596, 376 and 3,026 distinct domains, classes, relations, topic entities, and words, respectively.


(2014), but employs more flexible semiparametric density models.

we place a Gaussian process prior over densities that concentrates probability mass on densities that are close to the gamma family.


To overcome the limitations of existing methods, we propose a novel model, named Recursive Neural Conditional Random Fields (RNCRF).


The task is to learn a classifier to extract the set of aspect terms A i and the set of opinion terms O i from each review sentence s i ∈ S. This task can be formulated as a sequence tagging problem by using the BIO encoding scheme.

each review sentence s i is composed of a sequence of words s i = {w i1 , ..., w in i }.


The most important advantage of RNCRF is the ability to learn the underlying dual propagation between aspect and opinion terms from the tree structure itself.

as shown in Figure 1(c), where the aspect is food and the opinion expression is like.


(2015) showed that much of the performance gains of these new word embedding methods are due to certain hyperparameter optimizations and system-design choices.

if one sets up careful experiments, then existing word embedding models more or less perform comparably to each other.


In this paper, we approach the word embedding task from a different perspective by formulating it as a ranking problem.

given a word w, we aim to output an ordered list (c 1 , c 2 , • • • ) of context words from C such that words that cooccur with w appear at the top of the list.


2014, we use where we set x max = 100 and = 0.75 in our experiments.

we assign larger weights (with a saturation) to contexts that appear more often with the word of interest, and vice-versa.


Consequently, when rank (w, c) and hence ξ −1 w,c is large, ρ ξ −1 w,c is small.

the loss function "gives up" on contexts with high ranks in order to focus its attention on top of the list.


We follow the findings and design choices of GloVe and use a symmetric window of size win with a decreasing weighting function, so that word pairs that are d words apart contribute 1/d to the total count.

when the corpus is small (e.g., 17M, 32M, 64M) we let win = 15 and for larger corpora we let win = 10.


Here, we adopt an effective nested beam search strategy at this step.

edges are added between the current concept fragment and its preceding connected components by iterating through these components in a right-to-left order 3 using an inner beamsearch.


If the current word triggers multiple concepts, we first append each candidate concept to the partial graphs in the beam agendas[i−1], by using a component-wise beam search way (see section 3.1), and then pick B-best extended partial graphs by exploiting the features from both the concept level and relation level to compute the overall scores.

judging whether a word is a function word is an important and difficult task.


Since our incremental decoding for AMR parsing is an approximate inference, it is very natural to employ violation-fixing perceptron here for AMR parsing training.

we use an improved update method "max-violation" which updates at the worst mistake, and converges much faster than early update with similar or better accuracy.


Additionally, in our training algorithms, the implementation of the oracle function is rela-tively straightforward.

when the i-th span is processed in the incremental parsing process, the partial gold-standard AMR graph up to the i-th span consists of the edges and nodes that appear before the end position of the i-th span, over which the gold-standard feature vectors are calculated.


To find out, we compute metrics of user behavior using the data sample of 1000 users and 10 million posts described in Section 5.2.

we calculate (1) activity: a user's total number of posts, (2) breadth: the number of subreddits a user has posted in, (3) focus: the proportion of a user's posts that appear in the subreddit where they are most active, and (4) engagement: the average number of posts a user contributes to each discussion they engage in.


We find the coefficient of the B feature in the model is positively associated with dogmatism (p < 0.001).

engagement with a dogmatic comment tends to make a user more dogmatic themselves.


This demonstrates the power of query expansion.

the technique that explores external corpora to obtain the relevant documents works better than the method which simply uses top-ranked documents.


We hence present an advanced version of our algorithm with an improved arc score formulation and show that this simple algorithm provides high quality solutions to the graph-based inference problem (Section 4).

we implement the algorithm within the TurboParser (Martins et al., 2013) and experiment (Sections 8 and 9) with the datasets of the CoNLL 2006-2007 shared tasks on multilingual dependency parsing (Buchholz and Marsi, 2006;.


Due to the aforementioned conditions on the induced dependency tree, every arc that is added to T i−1 yields a set of lostArcs and lostParts that cannot be added to the partial tree in subsequent iterations.

the loss value is defined to be: every part that contains one or more arcs that violate the dependency tree conditions for a tree that extends the partial tree t i−1 ∪ {e} is considered a lost part as it will not be included in any tree extending t i−1 ∪{e}.


Our way to do this (Algorithm 2) is by estimating these probabilities for arcs and from these derive parts probabilities.

for the set of incoming arcs of a vertex v, E v = {e = (u, v) : e ∈ E}, a probability measure p e is computed according to: exp α×We e =(u ,v) exp α×W e Where α is a hyper parameter of the model.


Unlike modern approximation algorithms for our problem, our algorithm is greedy and deterministic.

in each iteration it selects an arc to be included in its final dependency tree and this decision cannot be changed in subsequent iterations.


Beam search In each iteration (lines 3-16) the algorithm outputs its |B| best solutions to be subsequently considered in the next iteration.

lines 4-10 are performed |B| times for each edge e ∈ E, one for each of the |B| partial solutions in the beam, b j ∈ B.


Post-search improvements After Algorithm 2 is executed, we perform s iterations of local greedy arc swaps.


We implemented our algorithms within the Tur-boParser (Martins et al., 2013) 2 .

every other aspect of the parser -feature set, pruning algorithm, cost-augmented MIRA training (Crammer et al., 2006) etc., is kept fixed but our algorithms replace the inference algorithms: Chu-Liu- Edmonds ((Edmonds, 1967), first order) and dual-decomposition (higher order).


The idea is to address both annotation styles simultaneously by sharing common feature representations.

johansson (2013) trained dependency parsers using the domain adaptation method of Daumé III (2007), keeping a copy of shared features and a separate copy of features for each treebank.


Given a sentence w (1:n) , the input layer builds a vector representation ⃗ r i w for each word w i based on both word and character embeddings.

an embedding lookup table is used to convert vocabulary words into their embedding forms directly.


The output layer employs a conditional random field (CRF) to infer the POS t i of each word w i based on the feature layer outputs.

the conditional probability of a tag sequence given an input sentence is: where Z(⃗ x) is the partition function: the output clique potential Ψ(⃗ x, ⃗ y i ) shows the correlation between inputs and output labels: Ψ(⃗ x, ⃗ y i ) = exp(⃗ s i ), with the emission vector ⃗ s i being defined as: where ⃗ θ 0 is the model parameter.


Figure 6 shows the effects of different corpus weights ratios.

a corpus weights ratio of 1:0.2 yields relative low accuracies.


We investigated two methods for utilizing heterogeneous annotations for neural network models, showing that they have respective advantages compared to their discrete counterparts.

neural stacking allows tighter feature integration compared to discrete stacking, and neural multi-view training is free from the feature and efficiency constraints of discrete one.


This paper proposes a context-aware pruning approach that can effectively solve the inefficiency problem of the coupled model, making coupled sequence labeling more generally applicable.

this work makes the following contributions: (1) We propose and systematically compare two ways for realizing context-aware pruning, i.e., online and offline pruning.


The basic idea is to consider only a small set of most likely bundled tags, instead of the whole bundled tag space T a × T b , based on evidences of surrounding contexts.

for each token w i , we only keep r one-side tags according to separate features f sep_a/b (.)


The correlations between POS tags are automatically captured in the learned POS embeddings and contribute to the improvement of parsing accuracy.

probabilities of grammar rules involving correlated POS tags are automatically smoothed in our approach without the need for manual features or additional smoothing procedures.


In our case, the neural network is able to capture the correlations between different POS tags as well as different valence values and smooth the probabilities involving correlated tags and valences.

our M-step can be seen as optimizing the ELL with a regularization term taking into account the input correlations.


It is a practical setting since many historical manually edited summaries of early streams are available and can be used as an example to demonstrate what kind of information is preferred in a stream summary.

3 Representing a text stream using Burst Information Network if a word w is in a burst state at every time t during a period, we call this period as a burst period of w, and w has a burst during this period.


We applied our BINet-based approaches (i.e.,  NodeRank and AreaRank) on the real-time stream.

we used news stream during Feb 5 to Mar 23 for training to generate news summaries for every day during Mar 24 to Mar 30 in an online fashion.


We present a new Convolutional Neural Network (CNN) model for text classification that jointly exploits labels on documents and their constituent sentences.

we consider scenarios in which annotators explicitly mark sentences (or snippets) that support their overall document categorization, i.e., they provide rationales.


In this work we adopt a particular view of sentence importance in the context of document classification.

we assume that documents comprise sentences that directly support their categorization.


We next run the feature map through an elementwise non-linear transformation.

we use the Rectified Linear Unit, or ReLU (Krizhevsky et al., 2012).


As a first step, we extend the CNN model above to explicitly account for document structure.

we apply a CNN to each individual sentence in a document to obtain sentence vectors independently.


However, in the RA-CNN model this is a weighted sum.

weights are set to the estimated probabilities that corresponding sentences are rationales in the most likely direction.


This model exploits the intuition that the risks of bias across the domains codified in the aforementioned Cochrane RoB tool will likely be correlated.

if we know that a study exhibits a high risk of bias for one domain, then it seems reasonable to assume it is at an elevated risk for the remaining domains.


Anecdotally, we found this has a greater effect than the other model hyperparameters, which we thus set after a small informal process of experimentation on a subset of the data.

we fixed the dropout rate at the document level to 0.5, and we used 3 different filter heights: 3, 4 and 5, following (Zhang and Wallace, 2015).


This differs from previous comparisons in the context of classifying shorter texts.

in previous work (Zhang and Wallace, 2015) we observed that CNN outperforms SVM uniformly on sentence classification tasks (the average sentence-length in these datasets was about 10).


While collaborative ltering (CF) approaches are one of the most successful methods for building recommender systems, their performance deteriorates dramatically under cold-start situations.

low prediction accuracy is observed for users/items with very few ratings.


Although transfer learning gradually gains popularity in handling the cold-start issue (Roy et al., 2012), most of them assume a homogeneous model where observations in both domains are of the same type.

to transfer knowledge to a ratingbased/text-based recommender system, the source system must also be rating-based/text-based.


We propose the idea of using the latent factors of the nearest-neighbors to estimate the unknown parameters in the distribution for a user.

the latent factors of the nearest-neighbors, {u l } l∈kNN(i) , are regarded as a set of pseudo observations to replace the unavailable data, (u (1) i , u (2) i , .


The similarity scores along with the list of nearest-neighbors are transferred to the target domain to form a set of weighted samples, D, which can be used to estimate the unknown parame- The main purpose of assigning a sample weight w l to each of the pseudo observations u l is that by doing so, users with a higher source-domain similarity to user i will have a larger impact on the estimation of the target-domain parameters (µ i , Σ i ).

with this model specication, the similarity between users is transferred across domains.


Therefore, in our coordinate ascent algorithm, different from the original PMF model, we update the user latent factors one by one.

all user latent factors are regarded as xed constants except for the one, u i , to be updated.


we are dealing with a one-class collaborative ltering (OCCF) problem (Pan et al., 2008).


Figure 3(a) shows the results when no sourcedomain information is available and thus no transfer learning is performed.

all models are provided only with the YouTube ratings and the YouTube corpus.


To demonstrate the advantage of transfer learning, we study the scenario where only source-domain text and target-domain ratings are available.

the YouTube corpus in the previous analysis is replaced with the Twitter corpus.


Users in the testing set are all completely coldstart users.

we are only provided the Twitter corpus when making prediction for these users.


The most important observation is that the highest AUC is achieved when K = 50 and T = 150.

the optimal number of topics is different from that of user latent factors.


In this paper, we propose a Convolutional Neural Network (CNN)-based model with probabilistic weighted average pooling to address speculation and negation scope detection.

our CNN-based model extracts those meaningful features from various syntactic paths between the cues and the candidate tokens in both constituency and dependency parse trees.


These studies have proved the ability of CNN models in learning meaningful features.

our CNN-based model extracts various kinds of meaningful features from the syntactic paths between the cue and the candidate token in both constituency and dependency parse trees.


Compared to the standard average pooling, each element in Cpavg has a weight depending on its value.

during computing Cpavg, the most active elements with the largest absolute values (i.e., the maximum and minimum values) play the leading roles, while those less active elements with smaller absolute values have less effect.


Naturally, we would like to ask whether it is possible to decode, or at least partially infer, a word's property from the neuron behaviours of its context words.

if the model process a verbal phrase 'spill over' or 'in the garden', will the property of the word 'spill', 'in' be combined with the following word and decodable from the model neuron activation behaviours towards the following word, or will the property of the word 'over', 'the garden' be primed by the previous word and decodable from the model neuron behaviours towards the previous word?


In this paper, we propose a statistical framework to solve and generate Chinese character riddles.

we learn the alignments and rules to identify the metaphors between phrases in riddles and radicals in characters.


While English words are strings of letters together, Chinese characters are composed of radicals that associate with meaning or metaphor.

chinese characters are usually positioned into some common structures, such as upperlower structure, left-right structure, inside-outside structure, which means they can be decomposed into other characters or radicals.


The weights of the features are trained with riddlesolution pairs.

in the riddle solving task, for the set of solution candidates, we hold that the original solution as the positive sample and others are the negative samples.


In NLP, Collobert et al (2011) used Convolutional Neural Networks to model the unary potentials.

for Recurrent Neural Networks, Lample et al.


Similar to , the outputs of the bidirectional RNN layer ω(x) are used to model the unary potentials of a linear chain CRF.

the NN based unary potential φ nn (y t ) is obtained by passing ω(x) t through a standard feed-forward tanh layer.


In this method, the pairwise potentials are modeled through a nonlinear Neural Network which is dependent on the current word and context.

the pairwise potential ψ(y t , y t+1 ) in equation 2 is computed by using a one dimensional CNN with 1-D filter size 2 and tanh non-linearity.


Such k-gram vectors are obtained by concatenating the vectors of the k consecutive words in the sentences (Nguyen and Grishman, 2015b;Chen et al., 2015).

the previous CNN models for ED only focus on modeling the consecutive k-grams.


In order to make it compatible with the previous work, we follow the procedure in (Nguyen and  to process the trigger candidates for CNN.

we limit the context of the trigger candidates to a fixed window size by trimming longer sentences and padding shorter sentences with a special token when necessary.


In order to overcome such limitation, we propose to model all possible non-consecutive k-grams in the trigger candidate, leading to the following set of non-consecutive k-gram vectors: The non-consecutive CNN model then follows the procedure of the traditional CNN model in Section 2.1 to compute the representation vector for classification.

the only difference is that the computation is done on the input set N instead of C. the convolution score set in this case would be S(N ) = {f t v : v 2 N }, while the aggregating score would be: Note that the maximum operation in Equation 2 only requires O(n) operations while the naive implementation of Equation 3 would need O(|N |) = O(n k ) operations.


We apply the same parameters and resources as (Nguyen and Grishman, 2015b) to ensure the compatible comparison.

we employ the window sizes in the set {2, 3, 4, 5} for the convolution operation with 150 filters for each window size.


(2011) † 68.3 Probabilistic soft logic (Liu et al., 2016) † 69.4 CNN (Nguyen and  69.0 DM-CNN (Chen et al., 2015) 69.1 B- RNN (Nguyen et al., 2016a) 69.3 NC-CNN 71.3 The most important observation from the table is that the non-consecutive CNN model significantly outperforms all the compared models with large margins.

nC-Cnn is 2% better than B- Rnn (nguyen et al., 2016a), the state-of-theart system that only relies on the context information within the sentences of the trigger candidates.


Previous studies have shown that the NLP models would suffer from a significant performance loss when domains shift (Blitzer et al., 2006;Daume III, 2007;Plank and Moschitti, 2013;Nguyen et al., 2015c).

if a model is trained on some source domain and applied to a different domain (the target domain), its performance would degrade significantly.


In this paper, we propose a method where, given training data in a related domain with similar (but not identical) named entity (NE) types and a small amount of in-domain training data, we use transfer learning to learn a domain-specific NE model.

the novelty in the task setup is that we assume not just domain mismatch, but also label mismatch.


The model also has significantly more freedom to adapt itself to the target types.

collapsing the two matrices simplifies the learning task and removes the constraints imposed by the pre-trained W s .


We do not include the parameters of the source class O when we initialise parameters of the NE types, while copying the parameters of the source class O to the target class O.

let o be the index of source domain class O.


We first obtained their learned event facet phrases and event expressions, most of which refer to general events.

then we followed their paper and identified two types of news articles that are likely to describe a civil unrest event by matching the obtained phrases to the English Gigaword fifth edition (Parker et al., 2011) we first consider news articles that contain a civil unrest keyword such as "strike" and "protest" 1 , then we identify an article as relevant if it contains a sentence where either two types of facet phrases or one facet phrase together with an event expression are found.


We observe that subevent phrases often occur in lists and we focus on leveraging such local cues to extract subevents.

we identify conjunction constructions that contain three or more verb phrases, each verb phrase obeys one of the following two forms: verb + direct object or verb + prepositional phrase.


For direct comparisons, we tested our subevents using the same test data and the same evaluation setting as the previous multi-faceted event recognition research by (Huang and Riloff, 2013).

they have annotated 300 new articles that each contains a civil unrest keyword and only 101 of them are actually civil unrest stories.


In this paper, we consider ZSL in the case where both visual and linguistic concepts are represented by Gaussian distribution embeddings.

our Gaussian-embedding approach to ZSL learns concept distributions in both domains: Gaussians representing individual words (as in (Vilnis and McCallum, 2015)) and Gaussians representing visual concepts.


Instead of using frozen Skip-Thought embeddings, we learn a task-aware ordered distributed embedding for sentences.

each sentence in the story is embedded X = (x 1 , .


To the best of our knowledge, this is the first work to collect human attention maps for VQA.

as described in Section 3, we collect ground truth attention annotations by instructing subjects to sharpen parts of a blurred image that are important for answering the questions accurately.


Our basic interface design consists of a "deblurring" exercise for answering visual questions.

we present subjects with a blurred image and a question about the image, and ask subjects to sharpen regions of the image that will help them answer the question correctly, in a smooth, clickand-drag, 'coloring' motion with the mouse.


Since there are there metrics, F1, UAS and LAS, possibly reported at the same time, we use the weighted average to consider the effect of all metrics when choosing the best model on the dev set.

we use F 1 for SRL, 0.5 × LAS + 0.5 × U AS for parsing, and 0.5 × F 1 + 0.25 × U AS + 0.25 × LAS for the joint task.


Would an existing Propbank facilitate the future task of creating treebanks for low-resource languages?

could the traditional order of first creating treebanks and then Propbanks be reversed?


The function captures the distance between the label for a user and her tweet.

this function measures the penalty for dissimilar labels for a user and her tweet.


Our goal was to explicitly model each actor's fluency in different languages, using a model with sim-ple, interpretable parameters that can be used to encode well-motivated assumptions about the data.

we want to bias the model towards the belief that actors typically speak a small number of languages, and encode the belief that all actors participating in a message are highly likely to be fluent in its language.


Therefore, we characterize each subreddit with multiple topic vectors.

we compute LDA topic vectors for each discussion thread in a subreddit, and learn 50 representative topic vectors for each subreddit via k-means clustering.


For thread-level feedback, we use karma points of the discussion thread itself; for the user-level feedback, we compute each user's subreddit-dependent k-index (Jaech et al., 2015), defined similarly to the well-known hindex (Hirsch, 2005).

a user's k-index k j in subreddit j is the maximum integer k such that the user has at least k comments with karma greater than k in that subreddit.


Its traditionally most studied function is that to encode prosody of oral speech, i.e., the prosodic rhetorical function; see, e.g., (Kirchhoff and Primus, 2014) and the references therein.

the comma is assumed to possess a strong rhetorical function (Nunberg et al., 2002).


X represents an input sentence, which <s> canadian prime … year <s> canada … nato input sentence headline is represented as a sequence of indicator vectors, whose length is M .

x i ∈ {0, 1} |V | , and whose length is L. Here, we assume L < M .


We can see that the proposed method, ABS+AMR, outperforms the baseline ABS on all datasets.

aBS+aMR achieved statistically significant gain from aBS (re-run) for ROUGE-1 and ROUGE-2 on DUC-2004.


According to this fact, ABS+AMR tends to successfully yield semantically important words.

embeddings encoded through the AMR encoder are useful for capturing important concepts in input sentences.


The comparison between ABS+AMR(w/o attn) and ABS+AMR (with attention) suggests that the attention mechanism is necessary for AMR encoding.

the encoder without the attention mechanism tends to be overfitting.


The plot shows that there exits a unique minimum and both constraints contribute to achieve a better likelihood compared to their unregularized counterparts (for which λ 1 = λ 2 = 0).

the regularization imposed by the differential of context and target embeddings g 2 contributes more than the regularization on the em-beddings Ψ and Φ separately.


In future work, by taking advantage of the promising results of Robust Gram, we intend to explore the model's behaviour in various settings.


While a detailed analysis is beyond the scope of this paper, we compared the likelihood of seeing antonyms in our results using a gold standard set of antonyms for biology, chemistry, and physics terms from WordNik (Wordnik, 2009).

we created an antonym set consisting of the 304 terms from the biology, chemistry, and physics categories in Wictionary for which at least one antonym is listed in WordNik.


In this paper, we empirically investigate a neural network method to learn features automatically for AES, without the need of external pre-processing.

we build a hierarchical CNN model, with one lower layer representing sentence structures and one upper layer representing essay structure based on sentence representations.


It is based on feature augmentation, incorporating explicit correlation into augmented feature spaces.

it expands baseline feature vector x to be Φ s (x) = (ρx, (1 − ρ 2 ) 1/2 x) and Φ t (x) = (x, 0 p ) for source and target domain data in R 2p respectively, with ρ being the correlation between source and target domain data.


Following common practice, we augment the dataset by referring to the database during training and test time.

we use the database to identify and anonymise variables (cities, states, countries and rivers) following the method described in Dong and Lapata (2016).


We also define for each node n of equation tree T , functions Span-Start(n) and Span-End(n) to denote the minimum span of text containing the leaves of the subtree rooted at n. We define them as follows: if n is a leaf An equation tree T is called projective iff for every node n of T , either Span-End(lc(n)) ≤ Span-Start(rc(n)) or Span-End(rc(n)) ≤ Span-Start(lc(n)).

the span of the left child and the right child cannot intersect in a projective equation tree 2 .


Our experimental results are quite satisfying and raise a few interesting issues.

it suggests that predicting equation parses using a pipeline of structured predictors performs better than jointly trained alternatives.


Even though likely syntactically attaches to contracted, a natural reading suggests that John contracted the disease is factual; the only bit of uncertain information is how (or when) he contracted the disease.

assuming that the author of statement (1) is truthful, event contracted occurred with AGENT John and THEME the disease, but the MANNER (or TIME) may not have been when a mouse bit him in the Adirondacks.


In order to alleviate the annotation effort, we focus on selected modal constructions.

we select verb-argument structures that have one ARGM-ADV or ARGM-MNR role, and that role is one of the following modal adverbs: certainly, clearly, definitely, likely, obviously, possibly, probably, surely, or unlikely.


Following previous work (Saurí and Pustejovsky, 2009; de Marneffe et al., 2012), we found it useful not to restrict answers to yes or no, but to allow for degrees of certainty.

we asked "Given the 3 sentences above, do you believe that the statement [potential interpretation] below is true?".


Importantly, on overage, modal constructions have 2.13 interpretations scored with high certainty, and 1.23 scored 3 or higher.

on average, our procedure generates over 2 interpretation that are either true or false, and over 1 interpretation that is true per modal construction.


Focus features characterize the dependency subgraph chosen as focus to generate the potential interpretation.

we include the number of tokens, word form and part-of-speech tags of the first and last tokens, and whether the focus occurs before or after the verb.


When executing the procedure to generate potential interpretations (Section 4.1) with predicted linguistic information, however, we are unable to generate all of them due to incorrect and missing syntactic dependencies.

352 of the 379 interpretations are generated (92.8%).


Since this data is now available, we seek to analyze current NLP challenges and extract dialectal language from online data.

we investigate dialectal language in publicly available Twitter data, focusing on African-American English (AAE), a dialect of Standard American English (SAE) spoken by millions of people across the United States.


Besides comparing aggregate statistics on mr(T ), it is also interesting to look at the sentiment of m h r(t i ) and m e r(t i ) for each tweet t i .

for every pair of = , we want to study the fraction of tweets in mr(T ) where m h r(t i ) has sentiment and m e r(t i ) has .


The odds of expressing a negative opinion over positive opinion in Hindi is between 1.5 and 6 (T mov exhibits a slightly different pattern but similar preference, T pol shows a stronger preference towards Hindi for negative sentiment), whereas the same for English is between 0.1 and 0.6.

english is more preferred Statistic m h r m e r pr(⊗|λσ; T CS ) 0.39 0.45 pr(−|λσ; T CS ) 0.22 0.14 pr(−|λσ; T CS )/pr(+|λσ; T CS ) 2.2 0.34 Table 6: T CS statistics for testing hypotheses Ia and IIa for expressing positive opinion, and Hindi for negative opinion.


3), we can estimate the preference, if any, for switching to a particular language while changing the sentiment.

research in socio-linguistics has shown that users often switch between languages when they switch from non-opinion (⊗) to opinion ({+, −, 0}).


In this paper, we present the Capsule model for analyzing documents to detect and characterize events of potential significance.

we develop a model based on topic modeling that distinguishes between topics that describe "business as usual" and topics that deviate from these patterns.


Finally, there are connections between Capsule and recent work on Poisson processes.

we can interpret Capsule as a collection of related discrete-time Poisson processes with random intensity measures.


We model each message with a bank of Poisson distributions 2 -one for each vocabulary term: The rate dv blends the different influences on message content.

it blends three types of topics, intended to capture "business-as-usual" discussion and content related to significant events.


As a result, each message captures a different mix of general diplomacy discussion, entityspecific terms, and recent events.

the Poisson rate for vocabulary term (  1 ; : : : ; A represent entity-specific strengths, while 1 ; : : : ; T allow time intervals to be more or less "eventful."


We can use the mean of the fitted variational distribution to explore the data.

we can explore "business-as-usual" content using the posterior expected values of the general topicsˇ1; : : : ;ˇK and the entity topics Á 1 ; : : : ; Á A , and we can detect and characterize events using the posterior expected values of the event strengths and the event topics.


Specifically, we can explore "business-as-usual" content using the posterior expected values of the general topicsˇ1; : : : ;ˇK and the entity topics Á 1 ; : : : ; Á A , and we can detect and characterize events using the posterior expected values of the event strengths and the event topics.

to detect events, we define an measure that quantifies the "eventness" of time interval t. we first compute how relevant each message d is to that time interval: m dt D f .t d ; t / EOE dt .


For our second baseline, we drew inspiration from previous work on event detection in the context of news articles, and focused on each time interval's deviation in term counts from the average.

we ordered the time intervals 1; : : : ; T for each simulated data set according to this measure: We added tf-idf term weights for our third baseline: Finally, we randomly ordered the time intervals for each data set to serve as a straw-man baseline.


9 We treated this list of events as "ground truth" and assessed how well Capsule and each of the baselines described in section 4.1 were able to recover them-or, in other words, how well the methods identify these eventful weeks, compared to more typical weeks.

we used each method to construct a ranked list of time intervals.


(2006), with only slight changes to push its performance as high as we can, producing a very strong baseline.

we extend it with highway layers and use Dropout as regularization.


The values for learning rate, learning rate shrinking and mini-batch sizes as well as context size are fixed once and for all based on insights drawn from previous work (Hai Son et al., 2011;Sukhbaatar et al., 2015;Devlin et al., 2014) as well as experimentation with the Penn Treebank validation set.

the learning rate is set to 0.05, with mini-batch size of 128 (we do not take the average of loss over the batch, and the training set is shuffled).


The experimental results for recurrent neural network language models, such as Recurrent Neural Networks (RNN) and Long-Short Term Memory models (LSTM), on the Penn Treebank are quoted from previous work; for Europarl-NC, we train our own models (we also report the performance of these in-house trained RNN and LSTM models on the Penn Treebank for reference).

we train LSTMs with embedding size k = 256 and number of layers L = 2 as well as k = 512 with L = 1, 2.


In this paper we focus on a class of LMs, which we will call mixture of distributions LMs (MODLMs; §2).

we define MODLMs as all LMs that take the following form, calculating the probabilities of the next word in a sentence w i given preceding context c according to a mixture of several component probability distributions P k (w i |c): Here, λ k (c) is a function that defines the mixture weights, with the constraint that K k=1 λ k (c) = 1 for all c. This form is not new in itself, and widely used both in the calculation of smoothing coefficients for n-gram LMs (Chen and Goodman, 1996), and interpolation of LMs of various varieties (Jelinek and Mercer, 1980).


Maximum likelihood estimation: n-gram models predict the next word based on the previous N -1 words.

we set c = w i−1 i−N +1 and calculate P (w i |w i−1 i−N +1 ).


These methods define a context-sensitive fallback probability α(w i−1 i−n+1 ) for order n models, and recursively calculate the probability of the higher order models from the lower order models: (3) To express this as a linear mixture, we convert α(w i−1 i−n+1 ) into the appropriate value for λ n (w i−1 i−N +1 ).

the probability assigned to each P M L (w i |w i−1 i−n+1 ) is set to the product of the fallbacks α for all higher orders and the probability of not falling back (1 − α) at the current level: Discounting: The widely used technique of discounting (Ney et al., 1994) defines a fixed discount d and subtracts it from the count of each word before calculating probabilities: .


Our goal is to produce a model that can play the role of the speaker S in RG.


Literal listener Given a description d and a pair of candidate referents r 1 and r 2 , the literal listener embeds both referents and passes them to the ranking module, producing a distribution over choices i.

p L0 (1|d, r 1 , r 2 ) = R(e 1 |e 2 , e d ) and viceversa.


We evaluate our model on the reference game RG described in the introduction.

we construct instances of RG using the Abstract Scenes Dataset introduced by Zitnick and Parikh (2013).


We now turn to using table information when scoring output words.

sentences which express facts from a given table often copy words from the table.


This was done in a purely bottom-up empirical manner, as opposed to using 'standard' evaluation criteria known from argumentation literature (Johnson and Blair, 2006;Schiappa and Nordin, 2013).

we split all reasons into several reason units by simple preprocessing (splitting using Stanford CoreNLP , segmentation into Elementary Discourse Units by RST tools (Surdeanu et al., 2015)) and identified the referenced arguments (A1 or A2) by pattern matching and dependency parsing.


In the next step, we removed reason units with wrong polarity.

all reason units labeled with C8-* or C9-* should refer to the more convincing argument in the argument pair (as they describe positive properties), whereas all reasons with labels C5-*, C6-*, and C7-* should refer to the less convincing argument.


Let P 1 (R B ) be reason unit targeting B in argument pair P 1 and similarly P 2 (R B ) reason unit targeting B in argument pair P 2 .

two reason units target the same argument in two different argument pairs (in one of them the argument is more convincing while in the other pair it is less convincing).


The second models is BLSTM extended with an attention mechanism (Rocktäschel et al., 2016;Golub and He, 2016) combined with convolution layers over the input.

the input em-  bedding layer is convoluted using 4 different convolution sizes (2, 3, 5, 7), each with 1,000 randomly initialized weight vectors.


Inspired by these recent work, for discourse parsing, we propose a model that aims to repeatedly read an argument pair and gradually focus on more fine-grained parts after grasping the global information.

we design the Neural Networks with Multi-Level Attention (NNMA) consisting of one general level and several attention levels.


An external short-term memory is designed to remember what has seen in the prior levels and guide the attention tuning process in current level.

in the first attention level, we concatenate R 1 0 , R 2 0 and R 1 0 −R 2 0 and apply a non-linear transformation over the concatenation to catch the general understanding of the argument pair.


Each element O in the feature map is computed by a filter denoted by f j (1 ≤ j ≤ M ) from the N -gram word sequences in t for a fixed integer N , as , where • denotes element-wise multiplication followed by the summation of the resulting elements (i.e., a Frobenious inner product of W f j and x i:i+N −1 ) and ReLU(x) = max(0, x).

we construct a feature map by convolving a text fragment with a filter, which is parameterized by weight W f j ∈ R d×N and bias b f j ∈ R. Note that there can be several sets of feature maps where each set covers N -grams for different N .


In contrast, the PR-  Table 3: Results of instance-wise evaluation for anaphoric and cataphoric sets curves for these two methods in Figure 5 show that our method obtained higher precision than Ouchi's method at all recall levels.

it got high precision in a wide range of recall levels (e.g., around 0.8 in precision at 0.25 in recall and around 0.7 in precision at 0.4 in recall), while the precision obtained by Ouchi's method at 0.25 in recall was just around 0.65.


HUME labels reflect adequacy, rather than fluency judgements.

annotators are instructed to label a unit as Adequate if its translation is understandable and preserves the meaning of the source unit, even if its fluency is impaired.


However, there are differences observed when we break down by node type.

we see a contrast between Czech and Polish, where the IAA is higher for atomic than for structural units, and German and Romanian, where the reverse is true.


(2013b), a method for learning vector representations of words based on their distributional contexts.

their model describes the probability of a token w i at position i using logistic regression with a factored parameterisation, where i+j is a vector encoding the context over a window of size k centred around position i, W is the vocabulary and the parameters V and U ∈ R |W |×d are matrices referred to as the context and word embeddings.


Whereas exposure bias results from training in a certain way, label bias results from properties of the model itself.

label bias is likely to affect structured models that make sub-structure predictions using locally-normalized scores.


Once the input sequence is encoded, seq2seq models generate a target sequence using a decoder.

the decoder is tasked with generating a target sequence of words from a target vocabulary V. words are generated sequentially by conditioning on the input representation x and on the previously generated words or history.


RNN decoders are typically trained to act as conditional language models.

one attempts to model the probability of the t'th target word conditioned on x and the target history by stipulating that p , for some parameterized function g typically computed with an affine layer followed by a softmax.


The complete model (including encoder) is trained, analogously to a neural language model, to minimize the cross-entropy loss at each time-step while conditioning on the gold history in the training data.

the model is trained to minimize − ln T t=1 p(y t |y 1:t−1 , x).


However, because finding the argmax sequence according to this model is intractable, we propose to adopt a LaSO-like (Daumé III and Marcu, 2005) scheme to train, which we will refer to as beam search optimization (BSO).

we define a loss that penalizes the gold sequence falling off the beam during training.


assuming distinct scores, t is the t'th token inŷ t−1 is the RNN state corresponding to its t − 1'st step, and where we have omitted the x argument to f for brevity.


Accordingly, we also found it useful to use a "curriculum beam" strategy in training, whereby the size of the beam is increased gradually during training.

given a desired training beam size K tr , we began training with a beam of size 2, and increased it by 1 every 2 epochs until reaching K tr .


As in the word-ordering case, we also experiment with modifying the succ function in order to train under hard constraints, namely, that the emitted target sequence be a valid parse.

we constrain the output at each time-step to obey the stack constraint, and we ensure words in the source are emitted in order.


As the alignments are constrained to be monotone, we can treat the transition from timestep j to j +1 as a sequence of shift and emit operations.

at each input position, a decision of shift or emit is made by the model; if the operation is emit then the next output word is generated; otherwise, the model will shift to the next input word.