Unlike these models, our dynamical filter is employed for interaction.

a filter generation function is proposed to capture the related intra and inter dependent information of a sentence pair.


The clinicians in our team suggested Past Medical History to be the most informative section of a clinical note, from which critical inferences can be drawn about the patient.

we segmented these notes into sections using a simple rule based program capturing the formatting of these section headers.


The main reason is that dominant instances in the WikiSQL training set have only one WHERE condition, as shown in Table 5, thus the model might not have memorized enough patterns for the other two limited-data groups.

the pseudolabeled instances generated by our SQL sampler and QG approach are more precious to the limiteddata groups (i.e #where =2 and #where≥3).


Adjectives tend to distribute more evenly between domain-specific words and domain-general words, thus discriminating them is harder (e.g., "outdoor", "wide" and "cooking" are domain-specific words while "minimum", "same" and "many" are domain-general words).

we take a statistical approach and only delexicalize adjectives that are unique to the domain (i.e., did not appear in the training set of any other source domain).


We observe that the description of a KB constant that appears in the logical form (Article) is often semantically similar to some word in the utterance ("paper").

we can obtain signal for the identity of a KB constant by solving an alignment problem: each slot can be aligned to words in the utterance that have similar meaning to that of the gold KB constant.


Naturally, in some cases a KB constant is not semantically similar to any utterance word (e.g., the relation Field in Figure 1), which we will mediate by using a global inference procedure (Section 3.5).

our goal is to learn a model that given an abstract utterance-logical form pair (x abs , z abs ) produces an alignment matrix A, where A ij corresponds to the alignment probability p(x abs j | z abs i ).


A central challenge is that no gold alignments are provided in any domain.

we adopt a "distillation approach", where we train a supervised model over abstract examples to mimic the predictions of an unsupervised model that has access to the full lexicalized examples.


Local Score Because inference is applied only at test time, we have access to the lexicalized utterance and not only the abstract one.

the aligner outputs a distribution over words for a slot y (e.g., in Figure 1, $REL DATE aligns with high probability to VERB, which corresponds to the word "published").


Intuitively, we would like to assign a KB constant that has high similarity with words the slot is aligned to.

we define s local of a KB constant c i for every slot y i to be its expected semantic similarity under the alignment distribution: We define sim φ (x j , c i ) to be the cosine similarity between the embedding φ(x j ) and the embedding φ(c i ) (scaled to the range [0,1]), where φ(c i ) is defined to be the average embedding of all words in L( Global Score Utilizing only a local scoring function raises several concerns.


For example, the logical form in Figure 1 contains the Field relation, however "field" is implicit in the utterance.

we define exe K (z) to be true iff z executes against K without errors, and define a global score that prevents assignments c that result in a logical form z such that exe K (z) is false.


In Section 2, we define the predicate span (p, p) as the NULL span, implying that we can regard the spans whose score is lower than that of the NULL span as an inappropriate argument.

we remove such tuples from the set of the candidates U.


For the second input, odd-numbered layers receive h Between the LSTM layers, we use the following connection (Zhou and Xu, 2015), Here, we firstly concatenate x t , and then calculate the inner product of the concatenated vector and the parameter matrix W (ℓ) with the rectified linear units (ReLU).

we obtain the input representation x (ℓ+1) t for the next (ℓ + 1-th) LSTM layer.


To do so, we use a script to remove (and later fill in) all the table/column names and value tokens in the labeled question-SQL pairs, and then group together the same SQL query patterns.

each SQL query pattern has a list of about 5-20 corresponding questions.


In our task, we want to generate more complex training examples.

we filter out simple SQL query patterns by measuring the length and the number of SQL keywords used.


For each slot, we also add column type or table information.

for example, columns with string type do not appear in the column slot with integer type during data augmentation (i.e., slot refilling) process.


Compared to our model which employs the coreference annotating mechanism, these two variants suffer notable loss in the precision of S metric.

their performance drops on the other metrics.


The training on unlabeled data proceeds otherwise in the same way as described in Algorithm 1.

the principle behind the MH (no feedback) model is the same as that behind self-training, i.e., a single model learns from its own predictions.


(2017a,b), we borrow two core insights of CxG: first, that morphemes, words, MWEs, and grammar are all on equal footing as "learned pairings of form and function" (Goldberg, 2013); and second, that constructions pair patterns of surface forms directly with meanings.

we can tag any surface realizations of constructions as meaning-bearing triggers (hence "surface construction labeling").


This metric is computed only over true positive connectives, as argument overlap cannot be evaluated automatically for false positives.

jaccard indices are not directly comparable between systems-they represent how well argument ID works given the previous stage, rather than in an absolute sense.


During the testing phase, the parser does not know whether cause-01 is a reentrancy node.

the parser needs to add all possible relation attachment actions (nearly half of the action space) into the valid action set as long as cause-01 exists in the stack, which makes the parser harder to train.


Therefore, the parser needs to add all possible relation attachment actions (nearly half of the action space) into the valid action set as long as cause-01 exists in the stack, which makes the parser harder to train.

we define several properties that the compact graph should respect to further refine the search space 4 : • Acyclicity: cycles are forbidden.


According to preliminary results (Table 3), we find that the performance of such an unsuper-vised model is not as good as the JAMR aligner when aligning relation and non-lexical concepts.

we propose a hybrid aligner, which combines unsupervised learning and rule-based strategy 6 .


Because our transition system is built based on the compact AMR graph, we could not evaluate the parser by isolating this representation completely.

we choose to remove some constraints defined on the compact AMR graph to investigate its effect.


Comparing the two sets of words associated with the idiom, we see that the literal set of words also tend to co-occur with just wind, a constituent word within the idiom.

even without annotated data or dictionary, we may still approximate a representation for the literal meaning of an idiom by the idiom's constituent words and their semantic relationship to other words.


We do not want to rely on any manual annotations for this step.

we integrate the automatically generated soft labels (based on MinV, our literal usage metric) into the unsupervised learning procedure as a weak form of supervision.


First, we show that ground-truth polysemy degrades performance in word similarity tasks.

word similarity tasks are not suitable as an evaluation test for polysemy representation.


To simplify the analysis, we further assume that X l = 1 ∀l.

the similarity between two words i, j can be plausibly measured (as customarily done) by the cosine distance between their contextual represen- the average distance is not equivalent to the distance between the average representations alone, but has to include an additional bias term cov(X i , X j ) -which reflects the statistical dependence between the two vector representations X i , X j .


Recall that all pairwise scoring metrics produce a positive score for adjective pair (j u , j v ) when it is likely that j u < j v , and a negative score otherwise.

the MILP approach should result in x u < x v when score(j u , j v ) is positive, and x u > x v otherwise.


Recall that random variables X and Y are independent if and only if the joint probability density P XY and the product of the marginals P X P Y are equivalent.

we can measure the dependence between random variables X and Y via the difference between P XY and P X P Y .


The mutual information (MI) 3 between two random variables X and Y is defined by (Cover and Thomas, 2006), where KL[• •] denotes the Kullback-Leibler (KL) divergence.

mI(X, Y ) is the degree of dependence between X and Y measured by the KL divergence between P XY and P X P Y .


Thus, MI(X, Y ) is the degree of dependence between X and Y measured by the KL divergence between P XY and P X P Y .

here, by definition of the KL divergence, MI can be represented in the form of the expectation over P XY , i.e., the summation over all possible pairs (x, y) ∈ X ×Y: The shaded part in Equation 5is actually the pointwise mutual information (PMI) (Church and hanks, 1989): PMI(x, y) can be thought of as the contribution of (x, y) to MI(X, Y ).


First, the maximum likelihood estimator of PMI add scores deduct scores Table 3: Comparison of estimators of PMI and PHSIC in terms of methods of matching the given (x, y) and the observed (x i , y i ) in D. PMI matches them in an exact manner, while PHSIC smooths the matching using kernels.

pHSIC is expected to be robust to data sparsity (Section 4).


With this mechanism, even for completely unknown pairs, it is possible to estimate the cooccurrence strength by referring to observed pairs through the kernels.

pHSIC is expected to be robust to data sparsity and can be applied to phrases and sentences.


Our goal is to encode text sequences t a and t b into counterpart-aware vectorial representations h a and h b .

while inferring the adaptive textual embedding for sentence t a , we propose reweighting the importance of each word in t a to explicitly account for its alignment with sentence t b .


As illustrated in Figure 2, given two input embedding matrices X a and X b , we first compute the affinity matrix A ∈ R M b ×Ma , whose elements represent the affinity scores corresponding to all word pairs between sequences t a and t b Subsequently, we compute the context-aware matrix for sequence t b as where the softmax(•) function is applied columnwise to A, and thus A b contains the attention weights (importance scores) across sequence t b (columns), which account for each word in sequence t a (rows).

matrix X b ∈ R d×Ma in (10) constitutes an attention-weighted embedding for X b .


As illustrated in Table 2, S-ACNN significantly outperforms S-CNN on both datasets, demonstrating the advantage of the filtergeneration module in our ACNN framework.

with only one convolutional filter and thus very limited modeling capacity, our S-ACNN model tends to be much more expressive than the basic CNN model, due to the flexibility of applying different filters to different sentences.


Unfortunately, for most of the documents on the Web, the time-stamp metadata is either missing or can't be trusted.

predicting creation time from document content itself is an important task.


In other words, this framework operates on distances between pairs of points calculated within each domain and measures how these distances compare to those in the other domain.

it requires a weaker but easy to define notion of distance between distances, and operates on pairs of points, turning the problem from a linear to a quadratic one.


we consider a general Bayesian learning setting where each w v is drawn from a prespecified prior distribution amounts to the special case when the prior is concentrated on the preset value.


The "future" module makes each prediction without the right context or the current token itself.

it works like a neural language model that, instead of predicting which token comes next in the sequence, predicts which class of token comes next.


Note that we do not add the suffix " PP" to a bridging anaphor representing time information, because such an anaphor is likely to have the same semantic type antecedent (see Example 2).

we use semantic similarity instead of relatedness to find its antecedent.


There is a direct analogy between such a set of random walks and a text corpus: each node corresponds to a word and the sequence of nodes visited during a random walk is analogous to a sentence.

any distributional model that takes as input this artificial "corpus" can generate multidimensional representations of the nodes in the graph.


for a target node n t , the objective is to predict all other nodes in the same context.

two vectors of the resulting space will be close if their corresponding nodes occur in topological proximity within the graph.


On the other hand, the effect of setting λ = 1 is less intuitive: Recall that, by construction, each textual node is connected only to entity nodes; that is, when the current node is textual, the next node will be always sampled from C n .

setting λ = 1 creates paths following an alternating pattern, where each entity node is followed by a textual node, which in turn is followed by an entity node.


Hierarchical concepts in these knowledge bases provide a natural way to categorize and locate instances.

the common simplification in previous work will lead to the following two drawbacks: Insufficient concept representation: Concepts are essential information in knowledge graph.


Relation set R can be formalized as R = {r e , r c } ∪ R l , where r e is an instanceOf relation, r c is a subClassOf relation, and R l is the instance relation set.

the triple set S can be divided into three disjoint subsets: , where i ∈ I is an instance, c ∈ C is a concept, and n e is the size of S e .


In this condition, the two spheres need to get closer in optimalization.

the loss function is defined as 2. s i intersects with s j (Figure 2c).


But these two datasets are not suitable for our model because FB15K mainly consists of instances and WN18 mainly contains concepts.

we use another popular knowledge graph YAGO (Suchanek et al., 2007) for evaluation, which contains a lot of concepts from WordNet and instances from Wikipedia.


We have made HyTE's source code and datasets used in the paper available at https://github.com/malllabiisc/HyTE Temporal fact and event extraction: Time, apart from being an information, also introduces a separate dimension to knowledge.

temporal scoping of relational facts is an imperative part of automatic knowledge graph construction and completion.


We observe that time is the main source of different many-to-one, one-to-many or many-to-many relations, e.g., (h, r) pair can be associated with different tail entity t at different points of time.

traditional methods fail to disambiguate them directly.


Furthermore, some important domainspecific words may not be present in the vocabulary of domain-general embeddings.

we have to regard such words as out-of-vacabulary (OOV) words, which may be harmful to the transfer learning process.


This is because two domain-specific embeddings are trained separately on source and target datasets, and therefore the two embedding spaces are heterogeneous.

model parameters trained in each model are heterogeneous as well, which hinders the transfer process.


Thus, we utilize the same word embeddings (general emb) for training both source and target models.

we remove the word adaptation layer (cd emb) in our approach under this setting.


This is partly because INIT and MULT do not explicitly preserve the parameters from source models in the constructed target models.

the transferred information is diluted while we train the target model with more data.


Usually, their friends are not included in a knowledge base but may also have accounts on the same platform.

for such entity mentions, linking them to an account that is also on the platform is more practical than linking them to a knowledge base.


Nonetheless, since we focus on entity linking in this paper, we only try to find as many mentions that may refer to Yelp businesses as we can, and then let the annotators decide whether to link this mention to a business.

we use the following two ways to find mentions and then merge their results.


As a location based social network platform, Yelp only contains businesses for different locations of franchises, not franchises themselves.

in these cases, we ask the annotators to link the mentions when they refer to a specific location of a franchise, but label them as "unlinkable" when they refer to a franchise as a whole.


Through the activities of the users on the platform, the users, mentions, reviews and businesses in Yelp-EL form a network where there are different types of nodes and edges.

we use Heterogeneous Information Networks (HIN) to model it, and then design meta-path based features to capture the relations between mentions and their candidate businesses.


They only used this information along with the name of each entity to perform entity linking.

their focus is very different from ours.


The extensive linguistic study is not available for a clinical domain to the extent it is for the general domain.

mL approaches are preferred over rule based approaches in the clinical domain.


However, such design cannot deal with multi-label tokens.

we customize the conventional CRF layer in LSTM-CRF (Lample et al., 2016) into a Fuzzy CRF layer, which allows each token to have multiple labels without sacrificing computing efficiency.


The key motivation is that, even the boundaries of an entity mention are mismatched by distant supervision, most of its inner ties are not affected, and thus more robust to noise.

we design a new Tie or Break tagging scheme to better exploit the noisy distant supervision.


However, both the original scheme and the conventional CRF layer cannot handle multi-typed or unknown-typed tokens.

we propose the modified IOBES scheme and Fuzzy CRF layer accordingly, as illustrated in Figure 1.


In the Tie or Break scheme, entity spans and entity types are encoded into two folds.

we separate the entity span detection and entity type prediction into two steps.


The distant supervision cannot decide whether it is a Chemical, a Disease, an entity of other types, two separate single-token entities, or (partially) not an entity.

in the FuzzyCRF model, we assign all possible labels for these tokens.


It is understood that semantic typing is a key component in many natural language understanding tasks, including Question Answering (Toral et al., 2005;Li and Roth, 2005) and Textual Entailment (Dagan et al., 2010(Dagan et al., , 2013.

the ability to type mentions semantically across domains and text genres, and to use a flexible type hierarchy, is essential for solving many important challenges.


Furthermore, we support other primitives, e.g., DBPedia or Wikipedia entries.

our system can be used across type taxonomies; there is no need to restrict to previously observed types or retrain with annotations of new types.


In order to run OTYPER on different datasets, we disabled original dataset-specific entity and type features.

among the open typing systems, our system has significantly better results.


In practice, we expect users to also provide background scenarios that are completely irrelevant to the input question.

we sample a negative scenario for each input question and rule text pair, (q, r) in our data.


In some cases, the question sentence does not include any entity, such as "Who died in the same place they were born in".

existing methods that e t c Topic Entity Answer Constraint Figure 1: The query graph structure of STAGG.


Given an intermediate state s during SQG generation process, we may transit to multiple subsequent states s by applying different operations.

we greedily select a subsequent state s with the maximum γ(s ), where γ() is a reward function taking the features and outputting the reward of corresponding state.


In machine translation and many other tasks, to compute the expected metric with respect to the posterior distribution is often intractable.

sampling methods are commonly used in MRT training.


2 The labeling procedure is under the supervision of the "Completeness" criterion (Sun et al., 2018), so the facts recorded information in the sentence as much as possible (only auxiliary information and relation between facts are omitted (Sun et al., 2018)).

the SAOKE dataset is a valid opendomain sentence-facts parallel dataset for both OIE and OIN tasks.


Note that the supervision signals P O (S|F * , Θ O ) and P L (F|S * , Θ L ) do not rely on any supervised parallel corpus.

similar to the application of dual learning paradigm on NMT task (Xia et al., 2016), it is theoretically possible to use unparalleled sentences and sets of facts to compute these signals.


Currently, the only available collection is the sets of facts provided by the SAOKE dataset, where the supervised information is available.

we implement our dual learning system in a supervised approach, which uses the reinforcement learning algorithm to optimize the Orator and the Logician.


They represent Q and A as syntactic trees enhanced with Q-to-A relational information, and apply TKs (Moschitti, 2006) to (Q 1 , Q 2 ) and (A 1 , A 2 ).

they model cross-pair similarity, and learn important patterns occurring in Q and A separately.


The weights in the Q and A convolution layers are learned by means of the backpropagation algorithm on the training Q/A pairs.

obviously, classifying a new Q/A pair is partially equivalent to performing the implicit cross-pair Qto-Q and A-to-A comparison.


For example in Figure 2(c), the domain type of the predicate government position is politician, which becomes the implicit type of the answer.

we can filter type constraints which are irrelevant to the implicit types, preventing semantic drift and speeding up the generation process.


The improvement on WebQ is smaller than on CompQ, largely due to the fact that 85% questions in WebQ are simple questions (Bao et al., 2016).

of combination, our approach significantly outperforms the vanilla SP+NN approach on CompQ by 1.28, demonstrating the effectiveness of our approach.


has a relation business/person/company, which is hard to decide without the information that Alfead Kahn is a person and Cornell-University is a company.

type features learned from entity type classification are proper a priori knowledge to initialize the relation extractor.


Meanwhile, all relations in Freebase are defined on head types and tail types.

we can construct datasets for type prediction tasks with the same dataset.


In this paper, we argue that the noise label problem in distant supervision is mainly caused by the incomplete use of KG information.

we propose a label-free distant supervision method, which supervises the learning of the embedding of sentence patterns by t h and entity types, instead of hard relation labels.


One advantage of the MRT-based method is that it can explicitly optimize a global sentence-level loss (e.g., F1 score) rather than local token-level losses.

it may catch more sentencelevel information in the training time and match evaluation metrics better in the testing time.


Finally, we remark that if we view MRT as a fine tuning step, it can be applied in any joint learning model based on building the joint distribution P (ŷ|s, θ) (e.g., the globally normalized P in (Zhang et al., 2017)).

we would think that MRT is a flexible and lightweight framework for the joint learning.


DU has the form of , in which w i , w i+1 are two adjacent tokens and d ii+1 is the dependency between them.

the low-dimensional forward and backward representation vectors of DU j are created by concatenating the corresponding final embeddings of tokens w j , w j+1 and the LSTM hidden state of the dependency d ii+1 .


Annotators were instructed to first read the section and then answer all questions.

annotators reveal possession information involving possessors and years that are potentially far away (different clauses, sentences, etc.).


the asteroid even though asteroid may only rarely be used to refer to similar entities in the general population.

rL varies less within a given dialog than across dialogs, and variation of rL has an inverse relationship with the length of the time two participants interact due to alignment of dialog participants' use of language (Clark and Wilkes-Gibbs, 1986;Garrod and Anderson, 1987;Brennan, 1996).


Secondly, we present a method for deriving a gradient (non-binary) measure of RA in situated dialog.

instead of first identifying REs and then resolving which entity they refer to, we treat all language in the dialog as being more or less referential, and use this gradual measure together with the referential semantics to derive which entity is being talked about.


However, one drawback to this method is that "the references must be contained in instructions that cause events involving the referents" and "it must be possible to automatically detect these events" (Schutte et al., 2011, 189).

rEs not referring to a detectable event cannot be detected in this manner.


Initial experiments showed that lemmatization did not affect the performance on our dataset.

each inflected lexical form is considered a unique word (i.e., vocabulary item).


Assuming that there are three state trajectories Then red states s 4 , s 9 , s 13 could be good candidates for "subgoals".

discovering subgoals by identifying hubs in state trajectories is equivalent to segmenting state trajectories into options.


Ideally, the vector softmax(o t ) in a well-trained SDN is close to an one-hot vector.

m • softmax(o t ) should be close to one column in m and we can view that m provides at most D different "embedding vectors" for RNN1 as inputs, indicating at most D different subgoals.


The former implementation takes a matrix of pairwise similarities between data points as input, whereas the latter approach is a generalization of k-means to an arbitrary matrix of pairwise distances.

they can be run on the scores relative to the question pairs (q i , q j ).


Finally, the clusters created by the LSSVM approach are more semantically related.

97% of all the suggested clusters contain questions with the same intent but, possibly, incorrect slots.


We did try a few pilot experiments where we setup a server to connect two AMT workers but we found that the probability of two workers simultaneously logging in was very low.

most workers logged in and left in a few seconds because no other worker joined simultaneously.


While there have been works trying to incorporate human design into cloze question generation (Zweig and Burges, 2011;Paperno et al., 2016), due to the expensive labeling process, the MSR Sentence Completion Challenge created by this effort has 1, 040 questions and the LAM-BADA (Paperno et al., 2016) dataset has 10, 022 questions, limiting the possibility of developing powerful neural models on it.

of the small size, human-created questions are only used to compose development sets and test sets.


If a question does not specifically test grammar usage, all of the candidate options would complete the sentence with correct grammar, leading to highly nuanced questions.

human-created questions are usually harder and are a better assessment of language proficiency.


In essence, LM treats each word as a possible blank and learns to predict it.

it receives more supervision than the LSTM trained on human-labeled questions.


Also, annotating unstructured EMRs requires a medical expert who can understand and interpret clinical text.

very few datasets like i2b2, MIMIC (Johnson et al., 2016) (developed over several years in collaboration with large medical groups and hospitals), share small-scale annotated clinical notes.


Hence, obtaining expert manual annotations in complex domains is infeasible as it is tedious to expert-annotate answers that may be found across long document collections (e.g., longitudinal EMR) (Lee et al., 2017).

we reverse engineer the process where we reuse expert annotations available in NLP tasks such as entity recognition, coreference, and relation learning, based on the information captured in the logical forms to populate entity placeholders in templates and generate answers.


The patient records in the smoking and obesity challenge datasets are categorized into classes with no entity annotations.

for questions generated on these datasets, the entire document acts as evidence and the annotated class information (7 classes) needs to be predicted as the answer.


(2) In emrQL-2, we first generate the instances of QL pairs from the templates and then distribute them into train and test sets.

emrQL-1 has more lexical variation between train and test distribution compared to emrQL-2, resulting in increased paraphrase complexity.


For example, in SQuAD (Rajpurkar et al., 2016) questions are designed to be answered given a single paragraph as the context, and most of the questions can in fact be answered by matching the question with a single sentence in that paragraph.

it has fallen short at testing systems' ability to reason over a larger context.


Second, existing datasets that target multi-hop reasoning, such as QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018), are constructed using existing knowledge bases (KBs).

these datasets are constrained by the schema of the KBs they use, and therefore the diversity of questions and answers is inherently limited.


As our ultimate goal is to solve multi-hop question answering, we focus on questions that the latest modeling techniques are not able to answer.

we constrain our dev and test sets to be hard examples.


However, Anne's belief about Sally's mental state does not match reality because Anne does not know that Sally has observed the change in the environment.

anne has a false belief about Sally's beliefs.


ing, the network will inevitably have to learn different weights and attention when presented with different orderings.

some traversal orderings may be easier to learn than others, leading to better (hopefully more semantic) abstractions.


However, m 5 has arity 0 and cannot have m 2 as its child.

it would be impossible for the RHT model to predict such a meaning representation as output.


Because of the unique indirect supervision setting of SpFD, the quality of the learned semantic parser is dictated by the choice of how to update the model parameters, often determined empirically.

several families of learning methods, including maximum marginal likelihood, reinforcement learning and margin based methods have been used.


When the search space is large, the policy may fail to sample a correct program, which can greatly slow down the learning.

off-policy methods are sometimes introduced to bias the sampling towards high-reward yielding programs.


For example, maximum margin reward penalizes only the most violating program and is benign to other incorrect programs.

the MMR algorithm could be unstable during training.


This is further corroborated from observing the much superior performance of off-policy policy gradient methods.

the sampling policy is an important factor to consider for policy gradient methods.


Policy Shaping vs Model Shaping Critique policy contains useful information that can bias the search away from spurious programs.

one can also consider making the critique policy as part of the model.


However, in our preliminary experiment, we found that Euclidean distance of unnormalized vectors gave poor performance.

following the common practice, we normalize all word embedding vectors to have a unit L2 norm in the construction of M (G) .


For top 100 mined pairs with the highest confidence scores (i.e., the score in Eq (3)), the accuracy is 98%.

it is easy to control the quality of mined pairs, which is important for a text mining algorithm.


ALA model can apply lemmas to encoding latent semantic relationship because lemmas share the information by multiple words and senses.

words sharing the same lemmas are likely to obtain the similar representations.


However, Fermi only evaluates on 675 of 2250 homographic contexts (Miller, Tristan and Hempelmann, Christian and Gurevych, Iryna, 2017) in SemEval2017 task7.

our model uses 675 as a test set and rest of data as a training set.


Currently, some mainstream approaches still focus on using unsupervised methods, i.e., language model based ones Tseng et al., 2015;Lee et al., 2016).

the development of CSC techniques are restricted and thus CSC performance is not satisfied so far (Fung et al., 2017).


(2012), there are only 398 syllables for thousands of characters in modern Chinese.

there are many Chinese characters sharing similar pronunciation, which further leads to the prominence of P-style errors.


For example, for the blurred image containing the character 领 (ling3), the Tesseract incorrectly recognizes it as 铈 (shi4), which is totally different from 领 (ling3) in shape.

these cases should be excluded because human are less likely to make such mistakes.


Consider that the quality of the training dataset has a significant impact on models' performance on the test datasets.

in this paper, we propose a metric, C train:test , to measure the correlation degree between the training and test dataset by calculating the number of spelling errors that occur in them: where E train and E test refer to the set of spelling errors in the training dataset and the test dataset, respectively.


Besides, we can see that the detection performance shows a stable improvement as the size of our generated corpus is continuously enlarged.

for data-driven approaches, it is of great importance to train our model with enough instances having different spelling errors.


We also believe that the tweets simply follow the standard language rules, and at some point eventually deviate from those.

tweets are a combination of formal expressions and informal expressions with conversions between different expression styles.


we use a low-dimensional context distribution vector t i as the input of the hyper LSTM to generate the weights of the main LSTM.

the weights of the main LSTM will be subject to a change in context vector.


Thus, the weights of the main LSTM will be subject to a change in context vector.

the main LSTM can predict the POS tag based on the different parameters.


Generally, Nĥ and N t are much smaller than N h and N x , respectively.

the size of the parameters in the hyper LSTM is hundreds of times less than that of the standard LSTM.


However, the values of the last three dimensions are close to zero.

we set this hyperparameter to 7 and we achieve a higher accuracy of 91.27%.


As reported in (Owoputi et al., 2013), if ARK-tagger does not add tag dictionary features and name list features, its performance will drop to 92.38%, which is lower than that of the DCNN.

our model is also competitive when lacking of knowledge.


The configurational information in sentences of a free word order language such as Sanskrit is of limited use.

the context of the entire sentence will be desirable even for basic processing tasks such as word segmentation.


In spite of considerable research on learning character embeddings, it is still not clear which kind of architecture is the best for capturing character-to-word representations.

a number of questions remain open: • Why is there a gap between methods for learning representations of sentences and words?


Formally, the sequence of labels is defined as: To define the scoring function f (h, y) for each position t, we multiply the hidden state h w t with a parameter vector w yt that is indexed by the tag y t to obtain the matrix of scores output by the bidirectional LSTM network.

the function f can be written as: In Equation 8, A is a matrix of transition scores, A i,j represents the score of a transition from the tag i to tag j, y 1 is the start tag of a sentence.


Also, learning word representations use fewer parameters than learning sentence representations.

the impact of training speed for sequence labeling is limited.


However, despite recent advances, the ability to recognize and understand actions in the real world is extremely challenging for artificial agents.

it becomes important for the agent to have an ability to explain and justify its action prediction.


We assume no prior linguistic knowledge.

translation approaches have to cope with the semantics and syntax of the language by discovering corresponding patterns in the data.


In the Figure, the attention map is depicted as a scaled and normalized 2D array of color codes.

each column in the array shows the attention distribution d t used to generate the predicted output at step t. each row in the array represents a triplet in the corresponding behavioral graph.


However, it turns out that when one baseline predictor -probabilistic context free grammar (PCFG) surprisal -is spilled over one position, the reported effects disappear: p = 0.816 for constituent wrap-up and p = 0.370 for dependency locality.

a reasonable but ultimately inaccurate assumption about baseline effect timecourses can have a dramatic impact on the conclusions supported by the statistical model.


The Knuth-Morris-Pratt (KMP) algorithm produces such a DFA (with O(n) states) in O(n) time (Knuth et al., 1977).

computing the infix probabilities of each of w's prefixes takes O(n(n|Q P |) m ) time, where m is the matrix multiplication constant 2 , as at each step one needs to rebuild the entire intersection automaton for the current prefix and compute an inverse on its transition matrices.


state q i with i < n + 1 has exactly one outgoing transition to state q i+1 and all other transitions are to states with labels at most i.

removing state q n+1 and all incoming and outgoing transitions and making q n the only final state results in the DFA for .


At stage k of the state elimination procedure on D, state q 0 is only connected to states up to label k − 1, thus α k−1 0,k+1 = ∅.

since we can simplify the expression as In addition, we recall and Algorithm 2 is for the offline setting-we know the entire string ahead of time and compute the infix probabilities of each of its prefixes incrementally.


For each element of the table we must perform one matrix addition, two matrix multiplications, and one matrix inverse on the matrices from the PFA, which has overall time complexity O(|Q| m ).

the runtime of Algorithm 2 is O(|w|(|w| 2 |Q| m )) = O(|w| 3 |Q| m ), where m is the matrix multiplication constant.


, L} such that the syntax path r(w) can be recovered from its embedding r(w) almost surely for sufficiently large K. This lemma (proof given in Appendix) suggests that as long as we choose a sufficiently large embedding dimension K, the above introduced encoding for syntax paths is essentially lossless.

for a given n-word sentence s = (w 1 , w 2 , .


For each of these cases, the distribution of C l can be made concentrated at its mean when K is made large enough (by invoking either the Weak Law of Large Number or the Central Limit Theorem).

if there exists a token in the path r(w) that makes Case 1 satisfied, the distribution of b is concentrated at 1, and our recovery scheme will detect, with probability close to 1, the token and its position in the path.


This presumes that each utterance is independent of the others, and clearly violates what we know about how language and conversation works, as discussed in the next section.

there have been many proposals to inject information from a longer context into standard LM architectures, going back to Ngram models (Bellegarda, 2004), or to generalize N-grams LMs to operate across utterance boundaries and speakers (Ji and Bilmes, 2004).


The problem with these approaches is that the modeler has to make design decisions about how to encapsulate contextual information as network inputs.

our approach here is to simply provide the entire conversation history as input to a standard LSTM-LM, and let the network learn the information that is relevant to next-word prediction.


Perhaps the most widely studied effect is topical coherence, or the tendency of words that are semantically related to one or more underlying topics to appear together in the conversation.

topic-related words are bound to re-occur across utterances, or certain related words appear to trigger one another (such as "children" and "school").


Linguistic conversation analysis has long noted that utterance types come in adjacency pairs (Schegloff, 1968), with preferences for certain pairs over others (like a statement is preferentially followed by agreement rather than disagreement).

words in an utterance should be more predicable based on the previous utterance.


Conversants also anticipate the ends of turns and jump in before the other speaker is completely done, making for very efficient use of time.

of all of these mechanisms, a good portion of conversations consists of overlapping (simultaneous) speaking.


In the reinforcement learning framework (Li et al., 2016c), the reward function used in the decoding considers the ease of answering, which is measured by a distance towards a set of 8 generic responses.

it can also alleviate the problem of generating generic responses to some extent.


Then, we find that MLE, which learns a 1-to-1 mapping in response generation, naturally puts more emphasis on optimizing the frequent patterns.

the converged local optimum is easy to output these patterns or their combinations, leading to generic responses.


1 will put the same emphasis on optimizing each of their loss terms.

l(C, θ θ θ) contains a large amount of patterns from the generic responses, thus it is not surprised to see that the trained models are stuck into local optimum that are inclined to generate these patterns or their combinations.


However, the PERSONA-CHAT dataset was created using an artificial data collection mechanism based on Mechanical Turk.

neither dialogs nor personas can be fully representative of real user-bot interactions and the dataset coverage remains limited, containing a bit more than 1k different personas.


the most difficult slot, will dramatically limit the overall model performance on the metric of the joint goal accuracy.

the initialization with a model pre-trained on the most difficult slot can improve the performance of the model on its weakness slot and boost the joint goal accuracy, while the initialization of a strength slot may not help much for the overall accuracy but in turn causes the over-fitting problem of the slot itself.


Tree substructures are rather important for task completion.

we introduce a third metric, Tree-Labeled (TL) F 1 , which compares the sub-tree structure for a non-terminal, instead of just the token span.


Concretely, in some countries, the mentor role is reserved for permanent/tenured faculty (for example, in Denmark).

a barrier to mentor standing for females can lead to an important under-representation of women.


4 Measuring Gender Biases Gender bias cannot be measured when evaluated on the original dataset as the test sets will follow the same biased distribution, so normal evaluation set will not suffice.

we generate a separate unbiased test set for each gender, male and female, using the identity term template method proposed in .


To remedy these problems, we propose methods to train a joint model across the multiple tag-sets of the different datasets, sharing statistical strength by using a single feature encoder across datasets while respecting the incompleteness of the labels during training.

our single model can take full advantage of all the available annotated resources and predict the full set of relevant types given a piece of text.


In order to label D datasets with some disjoint labels, we only consider the probability of the "observed labels" and allow the "unobserved" tokens to be free.

when tagging dataset i ∈ [D], we treat the non-entity tokens as potentially taking any entity type label from any of the other datasets as well as the 'O' label.


pv ← ∝ depth in tree 7: Bv ∼ Ber(pv) 8: if Bv then 9: ln ← ln + L(ŷv, y) 10: ifŷv > τ then 11: e ← 0 15: while e < epochs do 16: for each instance xi ∈ x do 17: hi ∆θ ← BACKPROP(li) 21: θ ← θ + α∆θ 22: e ← e + 1 return θ Incurring loss for all nodes along the path specified by y would place a disproportionate amount of emphasis on correctly applying terms that are 'higher' in the ontology, as loss will be propagated for the initial predictions concerning the application of these and then also, due to recursive application, for all of their children (and so on).

we only incur (and hence backpropagate) loss for a node v stochastically, according to a Bernoulli distribution B with parameter p v .


As in the case of recall, SD can be 'gamed': one can achieve a perfect score by predicting that all nodes apply to a given abstract.

this is only meaningful alongside complementary metrics like F1.


A token with a low HPE tends to attach often to the same part-of-speech, while a token with a high HPE will attach to many different parts-of-speech.

a language with a low HPE will tend to encode a lot of syntactic information in the morphology, rather than in word order say.


7 This can be seen as a problem with our datasets rather than the model itself.

an important future work in addressing the PCFP from an acquisition perspective is to create realistic and accurate data sets that model learner exposure both in word types and frequencies to enable assessment of the true difficulty of the PCFP.


Furthermore, unlike continuous image pixel values, words in a sentence are discrete tokens.

it is not possible to compute the gradient of the network loss function with respect to the input words.


Specifically, we first calculate the cosine similarity cos(•) between the vector pair V i and V j in different value subspaces, through the dot product of the normalized vectors 1 , which measures the cosine of the angle between V i and V j .

the cosine distance is defined as negative similarity, i.e, − cos(•).


Due to historical contact, there is much lexical overlap across Han logographic languages, as they borrowed words from one another (Rokuro, 1969;Miyake, 1997;Loveday, 1996;Sohn, 2001;Alves, 1999).

cognates in different languages are written using identical graphemes but pronounced differently.


The majority of logographs (characters) in Han logographic language family comprise of a radical that indicates its nominal semantic category and a phonetic radical that gives an inkling of the pronunciation (Defrancis, 1996).

patterns of co-occurrence of radicals across logographs might Figure 1: Geometric representation of the logograph "admire".


While in most languages, a syllable's pronunciation is influenced by neighboring syllables, most Han-logographic languages are monosyllabic and a logograph's pronunciation is rarely affected by neighboring logographs.

pronunciation prediction at the logograph (character) level for Han logographs is more appropriate.


To get this desired behavior, we employ regularization to maximize the likelihood of each token in the training data independently (as if reset every step).

we call this a unigram regularizer: and we add it to the training objective: the overall training loss is: L = L text + L U .


Standardly used LM models often have two layers of LSTM cells.

the issue we identifed might be mitigated in practice, since the hidden state of the first layer is not directly affected by tying the output and input transformation matrices.


We propose a method for incorporating linearized source parses into NMT that addresses these challenges by taking both the sequential source sentence and its linearized parse simultaneously as input in a multi-source framework.

the model is able to use the syntactic information encoded in the parse while falling back to the sequential sentence when necessary.


However, the parser operates at the word level.

we parse then break into subwords, so a leaf may have multiple tokens without internal structure.


However, the parser may fail on an input sentence.

we examine how well these systems do when given only unparsed source sentences at test time.


content words and function words) should differ.

we propose a new model with a mechanism called Self-Adaptive Control of Temperature (SACT) to control the softness of attention by means of an attention temperature.


In our assumption, due to the various functions of words, decoding at each time step should not use the identical attention mechanism to extract the required information from the source-side contexts.

we propose our Self-Adaptive Control of Temperature (SACT) to improve the conventional attention mechanism, so that the model can learn to control the scale of the softness of attention for the decoding of different words.


We define the new attention score and context vector asα andc, which are computed as: From the definition above, it can be inferred that when the temperature increases, the distribution of the attention score α is smoother, meaning that softer attention is required, and when the temperature is low, the distribution is sparser, meaning that harder attention is required.

the model can tune the softness of the attention distribution self-adaptively based on the current output for the decoder and the history of attention, and learns when to attend to only corresponding words and when to attend to more relevant words for further syntactic and semantic information.


Furthermore, sentences are translated in isolation.

pieces of information necessary to determine the gender of the speakers, might get lost.


In order to train a 2DLSTM unit, backpropagation through time (BPTT) is performed over two dimensions (Graves, 2008(Graves, , 2012.

the gradient is passed backwards from the time step (J, I) to (1, 1), the origin.


During the decoding, only the already generated target words are available.

either all 2DLSTM states have to be recomputed, or it has to be extended by an additional row at every time step i that cause higher complexity.


R: Darum, Frau Ashton, ist Ihre Aufgabe in diesem Zusammenhang von unschätzbarem Wert.

t: Mrs [NNP] table 2: Source, reference, template, and examples of template-token combinations.


We don't want too much information from TM to affect the translation of the source sentence.

we incorporate the gated TM in a way that it can only indirectly influence the target generation via hidden states.


However, the first problem is only found in European language datasets and the second problem occurs in all datasets but does not seem to bother pre-neural MT systems.

their explanations are not satisfactory.


As beam size increases, the more candidates it could explore.

it becomes easier for the search algorithm to find the </eos> symbol.


Also, the views might have different qualities.

we consider attentionbased weighted average or MULTIVIEW-ATT as an alternative to MULTIVIEW-CON.


(2016)'s image recognition and Pappas and Popescu-Belis (2017)'s document classification findings are similar.

not only does the increased amount of available information boost performance in the multiview setup, but also we can enable crossview transfer and learn a model that makes better predictions even if information is only available from a single view.


Our results show that while CVM and CCA based embeddings perform as well as the proposed embedding technique on semantic and syntactic tasks respectively, the proposed approach provides the best performance for both tasks overall.

this study demonstrates that existing bilingual embedding techniques are not ideal for code-mixed text processing and there is a need for learning multilingual word embedding from the code-mixed text.


At inference time, we will encounter only Latin script words, but we will not know the language of any token.

we use our jointrepresentation model to represent the token as both languages simultaneously.


20% (Bhat et al., 2018), but because those kinds of accuracies require supervised, in-domain training data, they are not directly comparable with the scenario that we are concerned with.

we compare against a natural baseline for our task: use a monolingual tagger and treat everything as if it were a single language.


Moreover, it's labor-intensive and time-consuming to annotate utterances of emerging intents and retrain the whole intent detection model.

it is imperative to develop intent detection models with the zero-shot learning (ZSL) ability (Lampert et al., 2014;Socher et al., 2013;Changpinyo et al., 2016): the ability to expand classifiers and the intent detection space beyond the existing intents, of which we have labeled utterances during training, to emerging intents, of which no labeled utterances are available.


The rich information conveyed by Chinese characters are capable of indicating semantic meanings of words even without correct orders (Qin et al., 2016;Wang et al., 2017b).

the CNN classifier, which encodes orders of words, may also bring such mis-order noise to classification.


However, infrequent or obscure labels will be easily confused or missed causing billing mistakes and/or causing the coders to spend more time annotating each record.

we believe methods that handle infrequent and unseen labels in the multi-label setting are important.


MIMIC III does not contain a standardized training/test split.

we create our own split that ensures the same patient does not appear in both the training and test datasets.


Unless a label appears at the top of the ranking, the annotator will not see it.

ranking metrics better measure the usefulness of our systems.


For zero-shot methods, the label vectors used are crucial regardless of the learning method used.

we evaluate ESZSL with three different sets of label vectors.


From the perspective of poetry education, the teacher will judge student-created poems in terms of some specific criteria and guide the student to cover the shortage, which naturally accords with DRL process.


3 Methods We apply our method to a basic poetry generation model, which is pre-trained with MLE.

we first formalize our task and introduce this model.


We expect moderate probabilities which fall into a reasonable range, neither too high nor too low.

we define the fluency reward of a poem P as: where µ and σ are the mean value and standard deviation of P lm calculated over all training sets.


It's quite intractable to quantify the meaningfulness of a whole poem, but we find that TF-IDF values of human-authored poems are significantly higher than values of generated ones (Figure 2).

we utilize TF-IDF to motivate the model to generate more meaningful words.


Direct use of TF-IDF leads to serious out-ofvocabulary (OOV) problem and high variance, because we need to sample poems during the training process of DRL, which causes many OOV words.

we use another neural network to smooth TF-IDF values.


Due to the limited amount of masterpieces, normal classifiers don't work well.

we use an adversarial training based classifier (Miyato et al., 2017), which achieves F-1 0.96, 0.73, 0.76 for the three classes respectively on the validation set.


Nevertheless, as discussed in Section 1, word-level likelihood or n-gram matching will greatly diverge from human evaluation manner.

we dispense with them and automatically evaluate generated poems as follows: Rewarder Scores.


For GT, we select poems containing the given words.


This may be due to nature of neural networks (Feng et al., 2017): during training, a neural network tunes its parameters to learn how to simplify different aspects of the sentence, which means that all the simplification rules are actually contained in the shared parameters.

if one simplification rule appears more frequently than others, the model will be trained to be more focused on it than the infrequent ones.


By taking a two-way advantage of both models, DMASS+DCSS takes a two-fisted approach to store the simplification rules in both additional and internal memory.

dMASS+dCSS achieves the best performance in SARI.


two elephants in an enclosure next to fit through a brick building we need to investigate a better method to compress the lattice.


This incorrect alignment would lead to meaningless lattice.

we introduce the semantic substitution matrix which measures the semantic similarity of each word pairs in context.


Unfortunately, this computation is exponential in the number of sequences.

we use iterative pairwise alignment which greedily merges sentence pairs (Durbin et al., 1998).


In our method, we use the state s t to keep track of the current confidence of target object represents the set of all the objects that can be chosen by the user.

the state s t is a probability distribution over all the objects and s t,i is the confidence that the object o i is the target object o tgt at time-step t. The initial state s 0 can either be a uniform distribution or initialized by the prior knowledge.


The reason is that as for the o tgt , question (a) is more relevant and can narrow down the searching space to a greater extent.

it is necessary to design a better reward function to estimate a non-zero immediate reward r t , and make the long-time return G t = T k=0 γ k r t+k+1 more informative.


Since most benchmark datasets randomly split the KG triples into train, dev and test sets, the queries that have multiple answers may fall into multiple splits.

some of the test queries (e s , r q , ?)


For example, if the input subset sources start with "p", we can learn to generate output subset targets that start with "p".

we exploit the input subset for learning idiosyncracies at test time and then generate the output subset using a modified model.


The training objective is to maximize the joint likelihood for the tasks of paradigm completion and autoencoding: where A is a set of autoencoding examples, e ✓ is the encoder, and D is a labeled training set of tuples of source s, morphological source tag t S , morphological target tag t T , and target w. MED was originally developed for morphological reinflection.

it operates on pairs consisting of a single source and a single target form.


The input x is either the output c of a CNN or h of a LSTM.

we can decompose x into β and γ contributions.


Hence, the contribution of the synthetic character becomes irrelevant and the model attributes the prediction to other characters.

we can conclude that whenever there is a clear character-level pattern, the model learns the pattern and the CD algorithm is able to accurately attribute it to the correct character.


However, {o,r} is the second most important character set for the BiLSTM model.

the BiLSTM model deemed the interaction between a root and suffix character more important than between two suffix characters.


Only 750 sentence received 5 reliable judgments.

to evaluate the quality of the results, the inter-annotator agreement was observed on those.


Moreover, the RA measure is extremely sensitive to the inconsistency of results, as the presence of one incongruous annotation out of 5 is enough to lower the outcome.

we used a second measure to take into consideration the difficulty of having 5 people agree.


In fact, in Table 1, the accuracy scores based on quantity-quality vectors (QQ column) suggest that our handcrafted features are actually useful for detecting hyperboles.

to gain further insight on their informativeness, we conduct a recurrent feature ablation and observed how different subsets affect predictions.


Building on top of the basic paragraph-level model, the CRF layer further improves the SE type prediction performance slightly by 0.4% and 0.7% in macro-average F1-score and accuracy respectively.

our full model with the CRF layer achieves the state-of-the-art performance on the MASC+Wiki corpus.


Adding the CRF layer further improves the performance of the paragraphlevel model on long paragraphs (with 10 or more clauses), while the influences to the performance are mixed on short paragraphs.

it is beneficial to model wider paragraph-level contexts and inter-dependencies between clauses for situation entity type classification, especially when processing long paragraphs.


In the equation above, the base intensity λ 0 (t) models the intensity of events that occur on their own initiative whereas αβe −β(t−s) models the intensity of events that are triggered by the previous event that occurred at time s. Here, multiplication of α and β represents influence of the previous event and β represents decaying rate of the influence.

the effect of the previous event exponentially decays with respect to the time difference.


Despite (1) scientific texts being inherently argumentative (Gilbert, 1976), (2) the existence of theoretical argumentative frameworks (Toulmin, 2003;Kirschner et al., 2015), and (3) a wide range of argument extraction models in other domains (e.g., debates or essays, see Palau and Moens (2009); , inter alia), there is still very little work on automatic argumentation mining from scientific literature.

there has been no work analyzing associations between argumentation and other rhetorical constructs in scientific writing, although such dependencies exist.


All of the tasks we consider are framed as multi-class classification tasks.

we simply specify all taskspecific losses to be L2-regularized cross-entropy errors.


For example, a time expression's parent can only be another time expression or a pre-defined meta node, and can never be an event; and an eventive event's parent is almost certainly another eventive event, and is highly unlikely to be a stative event.

we include the time expression and event type information predicted by stage 1 in this model as a feature.


For example, in the event "took a trip", "trip" is more representative than "took" and "a".

we add an attention mechanism (Bahdanau et al., 2014) over the Bi-LSTM output vectors in each multi-word expression to learn a task-specific notion of headedness (Lee et al., 2017): wherex i is a weighted sum of Bi-LSTM output vectors in span i.


(2016) and consequently the size of the generated random forests tends to be very large.

apart from keeping most of the hyperparameters as reported by Jain et al.


Unlike the state-ofthe-art approach, which treats an action as a sequence, our model uses an attention mechanism to integrate information from a set of comments.

the prediction of Q-value is invariant to the permutation of the comments, which leads to a more consistent agent behavior.


The optimal action of the agent at each step is the action that maximizes the Q-function, Q(s, a), which denotes the long-term reward of choosing action a in state s. In practice, we learn this Qfunction using a parametric function, Q(s, a; θ), where θ is the model parameter vector.

the predicted optimal action of the agent is the action that maximizes Q(s, a; θ).


Second, finding the action that maximizes Q(s, a; θ) requires the prediction of all N K possible actions, which is intractable.

we need a procedure to approximate the predicted optimal action from a combinatorial action space.


Although they tried to fix this problem by feeding randomly-permuted comments to the model, a different permutation of the same set of comments leads to a different Q-value prediction.

the agent may not consistently select the predicted optimal action.


In a nutshell, the attention mechanism outputs a weighted sum of the comment representations, where the weights are learned by a subnetwork to indicate the importance of each comment.

the action embedding is invariant to the permutation of the comments, which leads to a permutation invariant Qvalue prediction.


Since some of the posts and comments were deleted by Reddit, we remove all the deleted posts and comments from the datasets.

the statistics of our datasets are different from that in He et al.


However, the positive sentiment of "food", due to the word ""good", and presence of conjunction "though" determines the sentiment of "raw vegetables" to be negative.

aspects when arranged as a sequence, reveal high correlation and interplay of sentiments.


Simply promoting all words may adversely affect the generated topics.

we calculate the sentiment alignment of each word in a vocabulary based on its average cosine similarity to the words in a general sentiment word list.


Other neural network models also have the same problem for hyper-parameter tuning.

our strategy is to use the development set from the target domain to optimize 1 and 2 for one problem (e.g., we only do this on E!BK), and fix their values on the other problems.


However, when considering unsupervised domain adaptation, we should not presume the label distribution of the unlabeled data.

it is necessary to conduct experiments using datasets that reflect real-life sentiment distribution as what we did on setting2 and the largescale dataset.


Whats more, the review sentences are usually very short and colloquial, which makes it more difficult to match properly by a common WSD algorithm.

it is critical to construct more reliable contexts for aspect candidate matching.


Such a subgraph satisfies the following conditions: 1) it contains all the nodes matched from aspect candidates; 2) the total number of nodes in the graph is minimal.

the induced graph is a weakly connected DAG.


TripAdvisor), which most manually labeled 3-6 prominent aspects for rating, we set K as five.

we ask each annotator who are familiar with the domain to give 5 aspect terms which they think are most important for each category.


3 One prominent aspect can be expressed by different terms.

it is difficult to achieve a satisfied inner-agreement.


However, textual information in the native language where the rumor happened can be more helpful when it comes to ver-ifying worldwide rumors.

we not only indexed English webpages by searching Google with images but also included Chinese webpages via Baidu.


There are in total four agreement labels.

we have eight agreement features in total.


However, in this particular event, the textual information of those titles and tweets are semantically close.

models learned from English rumors can easily work on Chinese rumors, which is helpful for our transfer learning.


The weight on the inverse term only matters relative to the model sparsity, as that term cooperates rather than competing with the predictive accuracy term (because it almost never hurts accuracy to add more to the rationale).

we set λ 3 to 1.0 when we want to include the inverse term.


(2016), computing the exact posterior is in general intractable.

we posit a variational distribution q φ (h) and maximize  2: A broad overview of our MVDAM model depicting the three major components:a discriminator, an inference network and a prior and captures cues from multiple views of the news article.


While "fake news" are spreading primarily on social media, they still need a "home", i.e., a website where they would be posted.

if a website is known to have published non-factual information in the past, it is likely to do so in the future.


Fact-checking organizations have been producing lists of unreliable online news sources, but these are incomplete and get outdated quickly.

there is a need to predict the factuality of reporting for a given online medium automatically, which is the focus of the present work.


They found that "fake news" pack a lot of information in the title (as the focus is on users who do not read beyond the title), and use shorter, simpler, and repetitive content in the body (as writing fake information takes a lot of effort).

they argued that the title and the body should be analyzed separately.


However, this study only focuses on seven categories of cyberbullying while our dataset consists of 40 classes.

our classification task is much more fine-grained and challenging.


Although we use the weighted random sampler to alleviate the problem of the imbalanced dataset, repeating the very limited data instances (less than 3k) of a group during training cannot really help the posterior and prior networks to give a reasonable distribution that can generalize well on unseen inputs.

when the model comes into the instances in the testing dataset, the performance can be less satisfactory.


However, our main interest is in measuring the salience of frames in general, not merely focusing on the primary frame.

we also use our lexicons to identify the presence of any frames in a document.


While the English language dominates the internet, native English speakers are far outnumbered by speakers of English as a foreign language.

a vast amount of static and dynamic web content is continuously generated by nonnative writers.


However, as this feature is unique to the dataset used for the present work, it is hard to see it generalized to similar tasks that use other datasets, even in the context of UGC.

we report the results obtained with all features, with (Table 7) and without (   Summing up, we have shown that the challenging task of native language identification in the context of user generated content, where English texts are authored by highly competent nonnative speakers with as many as 23 native languages, can be accomplished with very high accuracy, as high as 86% when evaluated in-domain, and almost 79% out-of-domain (Table 7).


Among native users this reflects spelling variations between US and UK English.

the spell-checker marks the following forms, i.a., in New Zealand English: 'Organisation', 'Recognise', 'Realise', 'Criticise', etc.


Intuitively, the main subject of a right-branching sentence is described first (in the left part) and is followed by additional modifiers (in the right part) (Berg et al., 2011).

the left half of a right-branching sentence is more likely to possess a clearer structure pattern and thus lead to higher generation accuracy than in the right part, since the main subject is usually simpler and clearer than the modifiers that providing additional information about the subjec-t.


Note that the agent needs to take tens of actions (with the number depending on the length ofŷ) to generate a complete sentenceŷ, but only one reward is available for all those actions.

rL training is inefficient due to the sparsity of rewards, and the model updates each token in the training sentence with the same reward value without distinction.


Apparently, for the source-side monolingual data, the pseudo target reference y should have good translation quality.

for each source-side monolingual sentence, we use the N-MT model trained from the bilingual data to beam search a target sentence and treat it as the pseudo target reference y. Afterwardsŷ is obtained via multinomial sampling to calculate the reward.


The best trade-off between MLE and RL objectives in our experiment is α = 0.3.

we set α = 0.3 in the following experiments.


In practice, we solve the problem above by maximizing the first term with gradient-based optimization and early-stopping after only a few update steps.

in the low-resource scenario, finding a good initialization ✓ 0 strongly correlates the final performance of the resulting model.


There are roughly 30K parallel sentences, which is relatively small compared to text-only machine translation datasets that have millions of sentences such as WMT14 EN→DE.

we propose a new multimodal machine translation dataset called IKEA to simulate the real-world problem of translating product descriptions from one language to another.


However, we may not want to solely rely on only a few most important words.

to produce the initial hidden state of the decoder, we take a weighted average of the attention vector t and the mean of encoder hidden states: where λ determines the contribution from each vector.


These sentences not only describe the visual appearance of the product, but also the product usage.

capturing the connection between English → German English → French Method BLEU METEOR BLEU METEOR Imagination (Elliott and Kádár, 2017) 30.2 51.2 N/A N/A LIUMCVC (Caglayan et al., 2017) 31.1 ± 0.7 52.2 ± 0.4 52.7 ± 0.9 69.5 ± 0.7 Text-Only NMT 31.6 ± 0.5 52.2 ± 0.3 53.5 ± 0.7 70.0 ± 0.7 VAG-NMT 31.6 ± 0.3 52.2 ± 0.3 53.8 ± 0.3 70.3 ± 0.5  (Caglayan et al., 2017) 27.1 ± 0.9 47.2 ± 0.6 43.5 ± 1.2 63.2 ± 0.9 Text-Only NMT 27.9 ± 0.6 47.8 ± 0.6 44.6 ± 0.6 64.2 ± 0.5 VAG-NMT 28.3 ± 0.6 48.0 ± 0.5 45.0 ± 0.4 64.7 ± 0.4 5 Experiments and Results We evaluate our proposed model on three datasets: Multi30K (Elliott et al., 2016), Ambiguous COCO , and our newly-collected IKEA dataset.


in Question 2, makes tiny contribution to imply the sentiment polarity for the QA text pair.

a well-behaved network approach should consider the importance degrees of different [Q-sentence, A-sentence] units for predicting the sentiment polarity of a QA text pair.


In contrast to the mtl model, for trl models all parameters are transferred to the main task, not just parameters that represent shared knowledge.

we suspect the lower scores of the trl models with DIP2016 are due to overfitting on the vast number of samples which shape the parameters much more than the comparatively small SemEval corpus could.


Emotions are expressed in nuanced ways, which varies by collective or individual experiences, knowledge, and beliefs.

to understand emotion, as conveyed through text, a robust mechanism capable of capturing and modeling different linguistic nuances and phenomena is needed.


(2) Subject Words: In contrast, subject words or topical words are usually clustered together, i.e., many subject words are interconnected by the same connector words.

a coefficient is assigned to all nodes in G e and is computed as: where cl i denotes the average clustering coefficient of node i which captures the amount of interconnectivity among neighbours of node i.


As they stand, the patterns constructed in the previous step contain limited information relevant to emotion classification.

the patterns are enriched using continuous word representations so as to preserve semantic relationship between pat- Pattern Examples < cw, sw > stupid * , like *, am * < cw, cw, sw > love you *, shut up * < sw, cw, sw > * for * < sw, cw, cw > * on the < sw, cw > * <hashtag> terns.


There are two cases where this could be useful: limited data and limited computing resources.

to test for pattern consistency, we randomly selected several pattern sizes 15 and trained a random forest classifier using the eight emotions dataset.


Under Assumption 2.2, it must be the case that 8x 2 X There are |X | constraints but only d + 1 free parameters.

self-normalization is a nontrivial assumption when |X | d. In the case of language modeling, |X | = |V | k d + 1, where |V | is the vocabulary size and k is the length of the context.


Given a scoring function s(x, y; ✓) that satisfies assumption 2.1, we can derive a scoring function s 0 that satisfies assumption 2.2 by defining x 2 R is a parameter for history x.

we introduce a new parameter c x for each possible history x.


Input reduction perturbs the input multiple times and can expose deeper issues of model overconfidence and oversensitivity that other methods cannot.

it can be used to stress test the interpretability of a model.


Words in a phrase are syntactically and semantically related to each other.

it can be advantageous to learn a context-aware representation inside a phrase while filtering out information from outside the phrase using self-attention mechanism.


The structured tables and text descriptions of the dataset were originally crawled from different sources.

a non-negligible proportion of texts is in fact describing information outside the given table, such as historical records (e.g., "win streak").


Since the latent variables are meant to capture the hierarchical categorization of the script, we make the assumption that when a higher level script category (for example, z 0 ) is observed with the actual sequence of events (x), determining the immediate lower level category (z 1 ) is a deterministic operation.

similar to VQ-VAEs, we parameterize the individual factors of the posterior, q i (z i = k|pr(z i ), x), as: is an encoding function specific to latent z i and e ij is the jth value em-beddings for z i .


For example, the event (he ate food) gives evidence to the high level category of a restaurant script, while the more specific event (he drank wine) gives more evidence to the lower level category fancy restaurant.

during encoding, it makes sense to allow each latent to decide which parts of the input to take into consideration, based on its parent latents.


The main reason, according to our analysis, is that the syntax-based system may generate correct syntactic analyses for partial grammatical fragments in L2 texts, which provides crucial information for SRL.

syntactic parsing helps build more generalizable SRL models that transfer better to new languages, and enhancing syntactic parsing can improve SRL to some extent.


However, adding new intents or slots will change the predefined ontology.

developers need to extract additional dialog features based on new ontology.


|d| means turns of a specific dialog d ∈ D. To deal with unsupported user actions, A s will be a subset of A s .

the KL term in equation (3) can be defined as follows: As the original policy parameters θ are fixed, the loss function in equation 3can be rewritten as: This objective will transfer knowledge of the original system to the "student" at the turn level.


Despite the great progress those methods have achieved, it is still a challenging and open task for intent detection and slot filling.

we are motivated to design a powerful model, which can improve the performance of SLU systems.


Per-dialog accuracy counts the percentage of dialogs where every response is correct.

even if only one response is incorrect, this might result in a failed dialog, i.e.


For AirDialogue dataset, tokens occurred less than 10 times are eliminated but no tokens are removed for the synthesized dataset.

there are 5,547 tokens left the experiments.


Note there might be multiple optimal ground truth flights that have the same price and satisfy the customers' requirements.


(2017) employ a single factor for capturing these two properties, which limits the capability of distinguishing one property from the other.

when editing a sentence towards a given outcome, the sentence content is likely to be changed, which should be suppressed as much as possible.


p(X|Y, Z) is the probability of observing the sentence X given Y and Z, which is modelled by a decoder network D. the reconstruction loss can be rewritten as: where H is the cross entropy loss for the decoder.


Note that the preparation (discussed in Section 4.1) determines that a pair of pseudo-parallel sentences are very likely to differ in the outcome factors, denoted as h(y, y ).

by aligning the surface wording difference of two sentences in a pair and the difference in their outcome factors, we intend to improve the performance of the encoder E 1 for generating the outcome factor.


Finally, Y * is determined as follows: Note that in (Mueller et al., 2017), Y * is determined as arg max Y ∈C F (Y ), which does not consider an outcome target.

the revised sentence X * is generated from X 0 and Y * via the decoder D: the content of X 0 is preserved with Z 0 , and the expected outcome is achieved with Y * .


To analyze the reason, let us assume there is a sentence x in the training set.

the loss of reconstructing x from y and z is included in L rec .


With teacher forcing, the discrepancy between training and prediction (also referred to as exposure bias) can quickly accumulate errors along the generated sequence Ranzato et al., 2016).

the generator G θ is next fine-tuned using RL, where the reward is given by the evaluator.


While in RbM-IRL, the evaluator keeps updating during training.

keeping track of a baseline reward is unstable and inefficient.


end-to-end neural text generation has drawn increasing attention from the natural language research community (Mei et al., 2016;Lebret et al., 2016;Wiseman et al., 2017;Kiddon et al., 2016).


Through the encoder, at each step, the previous line L i−1 , current line L i and title T are represented as concatenated forward and backward 5 vectors Following previous work (Kingma and Welling, 2014; Zhao et al., 2017; Yang et al., 2017), we assume that the variational approximate posterior is a multivariate Gaussian N with a diagonal covariance structure q φ (z|x, c) = N (µ, σ 2 I).

µ and σ are the key parameters 6 to be learned, and they are computed by where W q and b q are trainable parameters.


Note that for classical Chinese poem, the dominant genres are quatrain and eight-line regulated verse with either 5 or 7 characters in each line.

our model is targeted to generate poems within these two genres, especially the quatrain.


In this way, the task requires the model to generalize well to both new SQL queries and new database schemas.

spider is distinct from most of the previous semantic parsing tasks because they all use a single database and have the exact same program in the train set and the test set.


More importantly, the number of logic forms or SQL labels is small and each program has about 4-10 paraphrases of natural language problem to expand the size of the dataset.

the exact same target programs appear in both the train and test sets.


Our task uses different databases for training and testing, evaluating the cross-domain performance.

spider is the only one text-to-sQL dataset that contains both databases with multiple tables in different domains and complex sQL queries It tests the ability of a system to generalize to not only new sQL queries and database schemas but also new domains.


However, they are able to generate nested and complex queries.

they can get a few hard and extra hard examples correct.


Each task has been given to three different raters.

each output has a separate score for each question that is the average of 3 different ratings.


We also observe that the number of question words is limited.

we introduce a specific vocabulary of question words to directly and explicitly model the generation of question words.


By optimizing the attention parameters, our model is expected to discover the correlation between target words and their relative distance from the answer.

distracted attention on irrelevant input words can be avoided.


Table 5: A bad case of baseline remains unresolved by applying answer-focused model because answer type is closely related to the context word "because" instead of the answer itself, but "because" is far from the answer.

the encoding of answer has little memory of "because".


Considering that the reward for different words in a sentence y t should be different, we further propose to use the reward at the word level as follows: It can be found that the classifier-based discriminator only provides reward for the finished sequence.

for a sequence of length T , to evaluate the action-value for a word at the time step t, Monte Carlo Search (MCS) with a roll-out policy G θ is usually applied to sample the unknown last T − t tokens (Yu et al., 2017).


Since the sentence-level reward reflects the novelty of the whole sentence, it gives repeated and short text low reward while novel and longer text high reward.

the generator is encouraged to generate novel text.


However, there is no explicit label about what style or category a poem or a sentence is for thousands of poems in the database.

traditional supervised sequence-to-sequence mod-els (Zhang et al., 2017) are not capable to generate stylistic poems without expert labeling.


The first sentences are randomly chosen from the poems in the test set.

we also consider the original poems written by real poets for comparison.


Note that the scores of human judgment are relative, not absolute.

we compare with seq2seq separately to avoid mutual interference.


In other words, the onehot style id and mutual information regularization cannot be torn apart.

the improvements over seq2seq all come from the part of style disentanglement modeling because seq2seq and our model share all the other components.


To the best of our knowledge, we are the first effort at stylistic poetry generation in a fully unsupervised manner.

our model requires no expensive expert style annotation for thousands of poems in the database.


Our proposed decoding objective constraints (topic and semantic) are complementary to the MMI objective, which encourages diversity and relevance to the source input.

in addition to comparing against the baselines described above, we evaluated three variants of our model: (1) maximum conditional likelihood combined with semantic and topic distributional constraints with a beam size of 10 (DC-10) (2) The same configuration with MMI-bidi re-ranking using a beam size of 10 DC-MMI10 and (3) MMI-bidi reranking with a beam size of 200 (DC-MMI200).


For example, Person is further divided into Athlete, Singer, Politician, and so on.

a template caption like 'Athlete celebrates after scoring.'


Using the template caption and image pairs (S, I) as training data, we regard the template caption generation as a regular image captioning task.

we adapt the encoder-decoder architecture which is successful in the image captioning task (Vinyals et al., 2015;Xu et al., 2015).


However, we want to test our model on real social media data and it is difficult to collect these informative captions for social media data.

we acquire training data from news outlets and testing data from social media.


(Tran et al., 2016) leverage face and landmark recognition to generate captions containing named persons, but such largescale training is difficult.

ooV words like named entities, which are a quintessential aspect of entity-aware captions because ooV words typically represent entities or events, are difficult to generate due to low training probabilities.


Our approach uses neural networks to generate dynamic templates and then fills in the templates with specific entities.

we are able to combine the sentence variation and surface realization quality of neural language modeling and the capability to incorporate novel words of template-based approaches.


Annotators were asked to succinctly describe all the differences in a short paragraph.

our novel dataset provides an opportunity to explore models that align language and vision, and capture visual salience.


The model is trained to generate a single sentence.

this model is similar to a typical captioning model but with soft attention over two images.


We see similar trends across other metrics.

object hallucination can not be always predicted based on the traditional sentence metrics.


This approach, however, neglects the correlation among target keyphrases since it does not model the one-to-many relationship between the document and keyphrases.

keyphrase prediction only depends on the source document, and ignores the keyphrases which have been generated.


Therefore, keyphrase prediction only depends on the source document, and ignores the keyphrases which have been generated.

the generated keyphrases suffer from duplication issue and coverage issue.


However, as explained in the introduction, most gradients are back-propagated through attention layer to the encoder's hidden state h t , not directly to the final memory state, and thus provide little incentive for the encoder to memorize salient information in c t .

to enhance encoder's memory, we add a closed-book decoder, which is a unidirectional LSTM decoder without attention/pointer layer.


As we can see, the closedbook decoder only depends on the word embeddings and encoder.

it can affect the entire model during training by influencing either the encoder or the word-embedding matrix.


The higher structuralcompression and structural-coverage scores will lead to higher ROUGE scores.

we can conclude that the structural-compression and structural-coverage regularization based on our hierarchical model have significant contributions to the increase of ROUGE scores.


Schluter 2017noted that only using the Rouge metric to evaluate summarization quality can be misleading.

we also evaluated our model using human evaluation.


Recent results have shown that better word representations can lead to significantly improved performance in sequence tagging tasks (Peters et al., 2017).

we first map each token w i into two embedding channels e (w) i and e (c) i .


Different desired lengths have different degrees of information attenuation.

the model is able to learn the probability of generating EOS with its own length information attenuation.


We assume reference summaries about the same topic to be semantically similar to each other, while summaries across topics are unrelated.

we created 2,160 pairs of similar summaries as postive data and 2,160 pairs of unrelated summaries as negative data.


In this experiment, the desired length is fixed for all the documents which is independent from the corresponding lengths of reference summaries such that the generated summaries may include more versatile words and phrases different from the reference summaries.

the similar-    " i thought i lost you , " jordan says .


The smaller p-value, the higher significant.

the difference of the similarity results is significant.


What's more, CNN enables much faster training and more stable gradients than RNN.

we select CNN seq2seq model as our basic model and do not compare our model with RNN seq2seq model.


Based on this understanding, we define a new expected reward function R APRIL for policy π as follows: where Note that ∆ x (y i , y j ) equals 1 if y j is preferred over y i and equals 0 otherwise (see §3.1).

r(y|x) is the relative position of y in the (ascending) sorted Y(x), and it can be approximated by preference learning.


This finding echoes with the training procedure illustrated in Figure 7, where more unlabeled data uniformly leads to better performance.

we believe that leveraging more unlabeled data for model training can boost model performance.


3) Images help users to grasp events while texts provide more details related to the events.

the images and text can complement each other, assisting users to gain a more visualized understanding of events (Bian et al., 2013).


Meanwhile, the coverage mechanism sums up all the historical attention distributions.

we regard the coverage vector as a global salience measure of the source being attended.


It has been shown that ROUGE correlates well with human judgments (Lin, 2004a;Owczarzak et al., 2012;Over and Yen, 2004).

we directly apply ROUGE to assess the salience of the text units.


A prerequisite for a pictorial summary to help users accurately acquire information is that the image must be related to the text.

we regard the image-text relevance as one of metrics to measure the quality of the pictorial summary.


In text generation studies, model selection by crossvalidation and model average with an ensemble decoder have been frequently used, but output selection as typified by majority vote has received less attention because of the fact that a majority output may not exist, as described above.

there is enough room to study this direction in the future.


Since SCU is the conceptual content of the text, it is difficult to automatically extract them from reference summaries by systems.

pEAK regards subjectpredicate-object triples as alternatives to SCUs and constructs a pyramid by clustering semantically equivalent triples.


Second, the score for a summary is determined by the correspondences between SCUs in the summary and those in the pyramid.

the score is defined as the sum of weights of SCUs that correspond to those in the pyramid in the summary divided by the sum of SCU weight possible for an average-length reference summary.


The pyramid method has two advantages over conventional manual evaluations: (1) the score is not intuitive but is systematically computed, i.e., the score can be explained as the sum of weights of SCUs in the pyramid, (2) the correspondences between the SCUs in a summary and the pyramid tell us whether the summary contains important SCUs or not.

the results explicitly tell us why a summary was given a good or bad score.


The major difference between PEAK and our method is that the former regards the reference summary as a set of subject-predicate-object triples while the latter regards a reference summary as a set of EDUs obtained from the source documents.

to construct high quality pyramids, PEAK is required to not only accurately extract the triples but also measure the semantic similarity between them accurately.


Actually, the F-measure of ClausIE is around 0.6 (Del Corro and Gemulla, 2013) and the correlation coefficients between the semantic similarity obtained from ADW and human judgment lie in the range of 0.55 to 0.88  (Pilehvar et al., 2013).

the resultant pyramids have insufficient quality to be practical.


The value of |S| m=1 s m is equal to the number of sentence whose EDU(s) is used in oracle summary.

an oracle summary that consist of fewer sentences tends to obtain a higher objective value.


In addition to this, even if another expression of time appears in other sentences, only one sentence (i.e., s 1 ) appears to be related to the question; thus, the system can easily determine the correct answer by attention, that is, by matching the words appearing both in the context and the ques-tion.

this kind of question does not require a complex understanding of language-e.g., multiple-sentence reasoning, which is known as a more challenging task (Richardson et al., 2013).


However, in some datasets, many questions have no textual overlap to determine the correct answer span in the context.

in order to avoid underestimating the baseline performance of those datasets, we used the GA system which is applicable to multiple choice questions.


Given that multiple-choice ques-tions offer multiple options that are of valid entity/event types, this gap was not necessarily caused by the limited number of candidate answers, as in the case with the answer extraction datasets.

we inferred that in the solved questions, incorrect options appeared less than the correct option did or did not appear at all in the context (such questions were regarded as solvable exclusively using the word match skill, which we analyzed in Section 4).


If these questions tend to require the word matching skill exclusively, attending the other portion is useful in studying a more realistic NLU, e.g., common-sense reasoning and discourse understanding.

we investigated whether these questions merely require word matching (see Section 4).


Settings: For each subset of the datasets, 30 questions were annotated.

we obtained annotations for 30 × 2 × 12 = 720 questions.


Our two heuristics can also be seen as the formalizations of these criteria.

to make questions more realistic, we need to create multiple hypotheses that require complex reasoning to be distinguished.


2, we obtain the path "between → hard → being → cottage → country" using our tree building method, which is not relevant to our question.

to improve the precision of useful concepts, we rank these knowledge paths by their relevance and filter out noise using the following 3-step scoring method: Initial Node Scoring: We want to select paths with nodes that are important to the context, in order to provide the most useful commonsense relations.


We approximate importance and saliency for concepts in the context by their termfrequency, under the heuristic that important concepts occur more frequently.

we score c ∈ {c 2 , c 3 , c 5 } by: score(c) = count(c)/|C|, where |C| is the context length and count() is the number of times a concept appears in the context.


We approximate this based on the heuristic that logically consistent paths occur more frequently.

we score this node via Pointwise Mutual Information (PMI) between the partial path c 1−3 and c 4 : Further, it is well known that PMI has high sensitivity to low-frequency values, thus we use normalized PMI (NPMI) (Bouma, 2009): score(c 4 ) = PMI(c 4 , c 1−3 )/(− log P(c 4 , c 1−3 )).


A large text corpus has high coverage, but the information is expressed using many different text patterns.

models which operate on these patterns (e.g.


Intuitively, combining layers would be more meaningful if different layers are able to capture diverse information.

we explicitly add a regularization term to encourage the diversities between layers: where λ is a hyper-parameter and is set to 1.0 in this paper.


In this paper, we are interested in evaluating the performance on long-range dependencies.

we focus on the subject-verb agreement category which consists of 35,105 instances.


In German, verbs must agree with their subjects in both grammatical number and person.

in a contrastive translation, the grammatical number of a verb is swapped.


The subject-verb agreement task is the most popular choice for evaluating the ability to capture long-range dependencies and has been used in many studies (Linzen et al., 2016;Bernardy and Lappin, 2017;Sennrich, 2017;Tran et al., 2018).

we also use this task to evaluate different NMT architectures on long-range dependencies.


However, these architectures perform well empirically according to BLEU.

we further evaluate these architectures on WSD, to test our hypothesis that non-recurrent architectures are better at extracting semantic features.


(2018); Domhan (2018) introduce the techniques in Transformers into RNN-based models, with reportedly higher BLEU.

it would be interesting to see if the same result holds true with their architectures.


This is because during decoding, all these beam search-  based systems must generate translation one word after another.

the parallelization advantage of the Transformer disappears.


Character-level NMT systems have a more difficult sequence-modeling task, as they need to infer the meaning of words from their constituent characters, where models with larger tokens instead delegate this task to the embedding table.

we hypothesize that increasing the model's capacity by adding layers will have a greater impact on character-level models.


We did not directly measure the difference in attention cost, but it cannot be greater than the difference in total cost for the smallest number of layers.

we can infer from Figure 3 that processing 5 layers in a character model incurs roughly the same time cost as attention.