Node clustering coefficient computes how many of the node's neighbors are directly connected among themselves.

if D is the number of neighbors of a node (equal to its degree), then the clustering coefficient is computed as 2×#related neighbors The nodes with a low clustering coefficient can be assumed to be more general, since their neighbors tend to be unrelated.


Our goal is to demonstrate that our feature set has good predictive power in such tasks, and not to build the highest performing classification system.

we experiment with a range of classification algorithms, and use them in their default parameter setup (without any tuning).


However, there has been a lack of analysis of the behavior of features across multiple datasets or using a range of classifiers.

it is difficult to determine which types of features will be most useful for a particular authorship attribution dataset.


Our study differs from previous work in that we perform dataset analysis using topic modeling followed by feature ablation experiments.

we are able to determine the level that each type of feature affects authorship attribution accuracy.


The relatively poor performance of the style features may be due to the baseline model (the continuous character n-grams) which effectively captured all the author's writing style.

the addition of auxiliary style features did not lead to any improvement.


The 100-billion-token corpus used in the original publication is not publicly available.

for the training of the LSTM models, we used the English Gigaword Fifth Edition (Linguistic Data Consortium (LDC) catalog number LDC2011T07).


In the WSD evaluation framework (Moro et al., 2014) that we selected for evaluation, these cases were either re-annotated or removed.

our F 1 on senseval2 cannot be directly compared with the F 1 in the original paper.


In addition, the ability to edit the entity pages easily with proper source citations and to view the edits reflected in the pages timely are some of the key reasons behind Wikipedia's growth and popularity.

wikipedia has become a destination of all kinds of information about entities and events.


However, all of these m references might not be appropriate to inherit.

the task here is to produce a ranked list of k references which A can inherit depending on the relevance of the reference with respect to the content of A.


There have been several reference recommendation systems (Huang et al., 2015;Caragea et al., 2013;Huang et al., 2014;Tang et al., 2014;Ren et al., 2014;Meng et al., 2013;Huang et al., 2012) for scientific articles but all of them either use the the text of the cited paper or the underlying citation network which makes these systems inappropriate to be used as baselines to compare with WikiRef.

we propose some standard baselines to ascertain the importance of each step in our approach.


The PorSimplesSent2, on its turn, is a middle term between the previous two: it has split sentences, but only the longest sentence derived from the split is paired with the original sentence.

we have chosen the dataset PorSimplesSent2 to be our gold standard for sentential complexity task in Portuguese.


Besides that, 80 pairs were dropped from our dataset because they contain nearly identical sentences or completely different sentences (improperly paired due to alignment error).

all the three totals in Table 6 were reduced by 80, resulting in 10,536, 4,888, and 2,520 sentences, respectively.


Those relevant words should be emphasized more while the irrelevant words should be less weighted during decision making.

some approaches (Parikh et al., 2016;Chen et al., 2017a) adopt attention mechanism to implicitly align the words between two passages to yield a better performance.


However, as shown in Figure 1, human compares H and P mainly based on Structure Analogy (Du et al., 2016;Gentner 1983;Gentner & Markman, 1997) instead of the merged meaning which will not only import irrelevant text but also lose information during merging.

only words with closely related syntactic/semantic roles (of the aligned predicates) should be compared.


However, we find the individual comparison here can describe the relatedness of those triplet-pairs better.

we generate the alignment weights using these individual comparison vectors with a self-attention mechanism as follows.


(2015) to delete those sentence pairs labelled with "-".

we end up with 549,367 pairs for training, 9,842 pairs for development and 9,824 pairs for testing.


Furthermore, we generate more precise alignment weights with these individual comparison vectors and self-attention mechanism.

our model is more interpretable than the previous models.


Since the structures of RSE and PSE are the same, it is necessary if the word representations for both languages have a shared vocabulary.

bilingual word embeddings (Shi et al., 2015) are employed to map words from different languages into the same feature space.


Corpus Gaokao challenge is an imitation of open-book examination for computers.

only quite a limited number of resources can be used, which makes common semi-supervised methods unlikely beneficial from large scale unlabeled data.


Labeling Examinees are required to understand the history knowledge from different sides.

the solver should also be capable of capturing such kind of features.


We build an SVM binary classifier for each DB field tag that judges whether the input target utterance chunk includes the information for that DB field or not.


To decide the values for these hyper-parameters, we randomly chose 200 utterance chunks from the training set of surrounding facilities to make a development set, and trained the model with the rest of the training set and evaluated it with the development set.

we found that λ 1 = 0.021 and λ 2 = 0.003 made the best performance on the development set.


Though deep neural models advance feature-engineering based charge prediction methods, they are unable to handle few-shot charges well due to the requirement of sufficient training data.

how to deal with these charges with limited cases is critical to a robust and effective charge prediction system.


1, many robbery case also contain the act of theft, and the existence of violence is the only key factor to distinguish these two charges.

how to capture the crucial factors for distinguishing confusing charges is another challenge of charge prediction.


However, when the attribute of a specific charge is NA, the label of the corresponding cases can be Yes or No.

we only add up the cross-entropy to the attribute loss when this attribute of the charge belongs to Yes or No.


Our method is characterized by the incorporation of attention mechanism and attribute-aware representations.

we design ablation test respectively to investigate the effectiveness of these modules.


The question, which is closest to the unseen question can be retrieved as a possible semantically similar question.

accurate semantic question matching can significantly improve a QA system.


An additional obstacle that needs to be overcome by natural language interfaces for databases, especially by data-driven approaches, is the strong dependence of the corpora on the schema of a given database.

even if one model is trained on a large dataset for a given database schema, it will not have good performance on different schemas.


A mononuclear relation holds a nucleus and a satellite, where the nucleus expresses the main textual information and the satellite offers additional information about the nucleus (Stede, 2008), while a multinuclear relation holds two or more discourse units, which are all nuclei.

three types of nuclearity exist: Nucleus-Satellite if the left subtree is the nucleus and the right subtree is the satellite, Satellite-Nucleus if the order of the satellite and nucleus is inverted, and Nucleus-Nucleus for multinuclear relations.


For example, in the case of a mononuclear relation, the nucleus unit is usually semantically closer to the topic of the paragraph.

our TMN model makes the semantic match between not only the different discourse units but also the discourse unit and paragraph, by three similarity metrics, namely, the Cosine, Bilinear and Single Layer Network, which can capture the features that are related to nuclearity recognition adequately.


Due to the need for our model to treat each word equally, Bi-LSTM is introduced to the Text Encoding module.

at each position , we concatenate the output ⃗⃗⃗⃗ , ⃖⃗⃗⃗⃗ ∈ ℝ of the two inverted LSTMs as the output of the current word as follows: each output contains not only the current word information but also the contextual information.


However, in our study, we find that the performance improvement of nuclearity recognition contributes to the improvement of the overall performance.

in this paper, we concentrate on these two tasks of macro discourse structure, structure identification and nuclearity recognition, and take structure identification as the main task.


As shown by Feng and Hirst (2012; 2014), for a pair of discourse units of interest, the sequential information from contextual units is crucial for determining structures.

it is well motivated to use CRF, which is a discriminative probabilistic graphical model, to make predictions for a sequence of units surrounding the pair of interest.


Since LSTM only considers the context from the previous, we utilize a bidirectional LSTM (Bi-LSTM) preserving both history and future information.

at each position t of the sequence, we can obtain two representations − → h t and ← − h t .


Due to the different reading order of two arguments, people may get the different focused information and thus have the different decisions.

we model the reciprocal attention on discourse arguments from two directions.


The performances are relatively stable surrounding 0.7, where all metrics under both settings are close to their peak values.

we exploit α = 0.7 as the final setting.


In particular, the model can achieve the best performances by using the MLP outputs.

we exploit the MLP outputs as the final implicit syntax features in our RST discourse parsing model.


The input to MAGE-GRU is no longer a sequence of words but a directed acyclic graph in which the word sequence is augmented with edges between arbitrarily distant words for which external knowledge suggests some explicit signals such as antonymy.

"loss" and "profit" in the example above are directly connected, making a downstream network layer more easily classify the discourse relation.


By the BPE operation, a word that appears frequently can be one unit, and rare or uncommon words can be expressed by the combination of several subword units.

we can express any words by the combination of small subword vocabularies to alleviate the out-of-vocabulary problem.


These two properties imply that SubW BPE naturally involves the methods using either character or word units in a single unified framework in terms of using subword units.

using SubW BPE , we are not required to distinguish the methods of using character or word units since they are respectively just a special case of SubW BPE with certain hyperparameters m = 0 and m = ∞.


2, these units are too small and lengthen the sentences.

computational time is also extended (see Table 6).


However, spelling and pronunciation of many languages (e.g., English, Japanese) can be ambiguous and inconsistent (Fushimi et al., 1999).

most transliteration systems are data driven and use context for disambiguation; e.g., (Yan and Nivre, 2016;Haizhou et al., 2004;Ekbal et al., 2006).


Our Wikidata datasets contain many multi-token names.

for constructing the train and test examples, we learn the word alignments of multi-token names, which is a simple task since the majority of name pairs contain the same word order, and we are not dealing with CJK languages with no token boundaries.


Industrial transliteration systems will contain dictionary look-ups for frequent named entity transliterations and employ automated methods, as evaluated here, to handle out-of-vocabulary cases.

it stands to reason that a fair evaluation of automated methods should include infrequent names even in the presence of some noise.


Recent works have shown that the target language-flag is weaker in a lowresource language setting (Lakew et al., 2017).

in addition to analyzing the behavior of bilingual and multilingual models, mainly, the zero-shot task requires a careful investigation.


To alleviate the global computations, we propose to convolute on the subgraph of adjacent mentions.

the overall coherence shall be achieved in a chain-like way via a sliding window over the document.


The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness.

we use joint embeddings of words and entities at sense level (Cao et al., 2017) to represent mentions and its contexts for feature extraction.


Deriving a mapping rule is nontrivial as semantic units in the structure can potentially span multiple tokens and matchers.

there are many ways to combine matchers and derive the rule.


What we found was that any threshold below 0.10 usually resulted in no auto-annotations, while a threshold above 0.20 resulted in many incorrect ones.

we chose 0.10, 0.15, and 0.20 for the FA, HFA, and UFA auto-annotation modes, respectively.


Moreover, for subsequent applications such as search engines or chatbot platforms to easily utilize the FG-NER results, we want to classify an entity to exactly one category in the hierarchy based on the context in which the entity appears.

we need an entity category hierarchy that does not allow overlap between the categories (Sekine, 2008).


This sentence must then be tagged as "<City>Tokyo</City> is the capital of <Country>Japan</Country>".

we have 20,000 sentences for English and 20,000 sentences for Japanese (as the number of leaf-level categories defined in Sekine's ENEH is 200).


After tagging the sentences with the top-level categories, we can convert the FG-NER problem into a simple classification problem (not a sequence labeling problem anymore), thus we can use SVM to classify the extracted entities at the top level into leaf-level categories.

we have a CRF model to tag the input sentences with top-level categories, and several SVM models (each for a top-level category) to classify the entities into the leaf-level categories.


Previous studies have shown that GloVe achieves the best performance for English NER task (Reimers and Gurevych, 2017).

we use the embeddings based on GloVe for English 5 .


Furthermore, the dictionary information can also be used in LSTM+CNN+CRF method.

we propose a method that efficiently utilizes dictionary information in the method LSTM+CNN+CRF and in both sequence labelling (CRF) and entity category disambiguation (SVM) phase of the method CRF+SVM.


This is because with a large number of character types (e.g., 2000 frequently used characters in Japanese), it is difficult to calculate high quality character embeddings.

we propose a modification to the existing LSTM+CNN+CRF network architecture to alleviate this problem: We remove the CNN layer from the LSTM+CNN+CRF+Dict method to have the LSTM+CRF+Dict method.


Because there are many Wikipedia entries that are in Japanese Wikipedia but they are not in English Wikipedia and we translated the dictionary from Japanese by looking up parallel entries in English Wikipedia, these entities do not appear in the English dictionary.

the F-score of English Rule+Dict is very low.


For FG-NER definitions that are not based on existing knowledge bases, it is difficult to automatically extract the NE tagged data from a large text corpus like Wikipedia.

for FG-NER system working with these definitions, we must design an algorithm that requires only a small amount of training data.


For example, an order-1 model assigns a very low probability to the uni-label "I" of the t-th word, which means the order-1 model is confident that the t-th word can hardly be labeled as "I".

it is unnecessary to take account of the bi-gram labels "I-B", "I-I", and "I-O" at the next time step.


The tags in consecutive segments may have dependencies, too.

we conduct experiments on the chunking and NER tasks to evaluate the proposed method.


However, most low-order tags have been assigned very low probabilities by loworder models and they will form almost impossible high-order tags.

we only keep a small subset of all low-order tags, which makes the possible combinations shrink rapidly so that the cost of dynamic programing is greatly reduced.


The automatic summarization problem has been studied since the middle of the 20 th century.

the application of several summarization approaches, such as statistical Alguliev et al., 2013;Rautray and Balabantaray, 2017) and graph-based (Erkan and Radev, 2004;Shen and Li, 2010;Mosa et al., 2017b) approaches, have been described in the literature.


Let D be a set of input documents to be summarized and each of these document is split into sentences.


Regarding the summary length constraint, each ant keeps its own length and stops when it reaches the maximum summary length.

mDS-ACS assigns a state for each ant; active ant if it can include more sentences to its summary and inactive ant otherwise.


We prioritized the speed of annotation over detail, opting for a 3-point scale rather than e.g., the 5point scale in SemEval Twitter datasets (Rosenthal et al., 2017).

the task was to rate the prevailing sentiment in complete posts from VK on a three-point scale ("negative", "neutral", and "positive").


8 A user may end a post with a "hedging" emoticon just to express friendliness or politeness, and we found that often to be the case for VK data.

emoticons were not taken into account when no sentiment was expressed verbally or when they aligned with the content of the message.


Gutmann and Hyvarinen (2012) proved that S(m) ≤ S(pce) for every matrix m. The rank of the matrix m is at most d + 1.

the NCE training goal is finding the best low-rank decomposition of the PCE matrix in the sense that it minimizes the difference S(pce) − S(m).


Attention, which is widely applied to Computer Vision (CV) and NLP fields, is an effective mechanism and has been demonstrated in image recognition (Mnih et al., 2014), machine translation Luong et al., 2015) and reading comprehension (Hermann et al., 2015;Cui et al., 2016).

some researchers have designed attention networks to address the aspect-level sentiment analysis and have obtained comparable results, such as AE-LSTM , ATAE-LSTM  and IAN (Ma et al., 2017).


The forward L-STM handles the sentence from left to right, and the backward LSTM processes it in the reverse order.

we can obtain two hidden representation, and then concatenate the forward hidden state and backward hidden state of each word.


IAN realizes the importance of interaction between aspect term and context, and models aspect term and context using two connected attention networks.

iAN performs better than ATAE-LSTM, and achieves an improvement of 1.40 points and 3.40 points on Restaurant and Laptop datasets in Three-class respectively.


Compared with ATAE-Bi-GRU, the most difference is that PAN utilizes the position embedding to calculate the attention weights rather than the aspect term embedding like ATAE-Bi-GRU.

according to these three experimental results, we can prove the importance of the position information in aspect-level sentiment analysis task.


For instance, in this case, PBAN should pay more attention on the word "take-out".

pBAN is capable of figuring out the important part in a sentence for judging the sentiment polarity by modeling the mutual relation between sentence and different words in aspect term.


susan was happy and sent each friend a thank you card.


The degrees of other (unshown) possible alignments for these answers are not large.

they are likely to be incorrect answers.


Example 6 Consider two adjacent sentences Each friend had a present for Susan.


Reading comprehension tasks also involve other questions.

gTCR is to be combined with other methods.


This can be explained by the fact that the corpora used to train the Amazon embeddings contain many more unlabeled reviews for books and electronics (973,194 / 122,438 respectively) than electronics and kitchen (21,009 / 17,856).

the vector representations for sentiment words that only appear in the books and DVD subcorpora are of higher quality than those that only appear in the electronics and kitchen subcorpora (see Table 4).


Working only monolingually is problematic, however, because AM is a difficult task even for humans due to its dependence on background knowledge and parsing of complex pragmatic relations (Moens, 2017).

acquiring (high-quality) datasets for new languages comes at a high cost-be it in terms of training and/or hiring expert annotators or querying large crowds in crowd-sourcing experiments.


The translators took about 40min on average to translate one essay and indicate the argument structures.

they required about 270 hours to translate the whole PE corpus into German, and the resulting overall cost was roughly 3,000 USD.


We choose 10% of the training data as dev set.

we have 286 essays in the train set with a total of 105,988 tokens (en) and 36 essays in the dev set with a total of 12,657 tokens (en).


We attribute this huge drop not to the larger language distance between English and Chinese (relative to English and German), but primarily to the domain gap between student essays and hotel reviews.

in fact, we observe that, e.g., major claims in PE are almost always preceded by specific discourse markers such as "i believe that" or "in the end, i think" (cf.


Highway network (Hinton et al., 2012) has been shown to bring significant improvements to many NLP tasks, We apply a 4-layer highway network to u, and a single layer highway network is calculated as: where g is called as transform gate, (1 − g) is called as carry gate, and g = sigmod(W g u + b g ) .

the equivalent word embedding x c is finally worked out by a linear projection: x c is the equivalent word embedding calculated by CharCNN, which has the same dimension as x w obtained by looking up V e w ∈ R |V e w | * dw .


Hence, the message sequence could be represented as Hence, we finally obtain a sequence Y h without OOVs, and HL-EncDec let the hybrid-level Y h to replace the word-level Y .

hL-Decoder is able to generate any word, no matter this word has been recorded in Decoder's vocabulary or not.


Though the automatically generated datasets usually consist of a large quantity of labeled data and make it possible to train large neural network models, they are in nature far away from real-world language understanding problems and have serious ambiguity issues .

the state-of-the-art system of cloze test almost reaches the performance ceiling and loses its improvement direction due to the limitation of the corpus .


As we know, CLOTH and RACE consists of passages designed for high school students, while Wikipedia entries are for the general public and therefore have a different word distribution.

how to make use of huge unlabeled data to help training is a key for performance improvement, since training corpora of higher quality are generally smaller in scale.


Unlike nouns, no direct supervision is available from WordNet because it does not have hyponym taxonomies for adjectives.

we use simple and conservative heuristics to detect adjectives as events if they are originated from present and past participles of verbs, illustrated as follows: (13) There is a man-made river in the country.


Adverbs connect with their modifying verbs, forming a single event nugget, as illustrated in (5).

we combine eventive adverbs with such verbs to detect resulting verb phrases as events.


From the perspective of open-domain event detection, supervised phrase detection models are likely suboptimal because they might be limited to particular domains or overfitting to small datasets.

we explore a simple dictionary-lookup approach to detect WordNet phrasal verbs as events, inspired by Yin and Schütze (2015).


As seen in the self-training model by Liao and Grishman (2011), erroneously generated training data worsen system performance.

we need to generate training data as accurately as possible.


One difference from traditional sequence labeling problems is that our output includes discontinuous phrases.

we leverage an extended version of the standard BIO tagging scheme, inspired by Metke-Jimenez and Karimi (2016).


Another 6% of the pairs are such that both metrics do not provide the correct judgment.

rOUGE and our SVr appear to agree on a large majority of the system pairs.


If we make a query 6 in these repositories for Spanish corpora annotated with negation, we only find 1 resource in ReTeLe catalog, the SFU Review SP -NEG corpus (Jiménez-Zafra et al., 2018).

we decided to perform an exhaustive search of resources.


The existing annotation schemes are not compatible because of differences in genre, annotation guidelines, and the aspects of negation that have been annotated (negation cue, scope, event, focus).

it would be desirable to define a new scheme that integrates the contents of existing schemes.


If these aspects could meet the users' requirement, they can give a high overall rating, otherwise, they could give low scores.

overall rating can partly reflect the user's attitudes to aspects, which is called overall rating prior.


The analysis proves users and overall ratings are significant on interpreting the sentiments of different aspects.

we introduce these two kinds of information into HUARN and detail the model here.


For example, out of the 14,000 words annotated with emotions in the NRC lexicon, almost 8,000 words (57%) are associated with more than one emotion.

we relax the labeling scheme followed in EWE UNI and design EWE MULTI to take into consideration a multi-class, multi-label setting, where a document can have more than one emotion label.


Therefore, we relax the labeling scheme followed in EWE UNI and design EWE MULTI to take into consideration a multi-class, multi-label setting, where a document can have more than one emotion label.

unlike EWE uNI , in EWE MuLTI the set of all emotions with a j (d) > 0 for document d is used as final emotion labels for d. the multi-label annotated dataset D is Our large corpus of unlabeled documents is extracted from the Amazon reviews dataset (McAuley et al., 2015) consisting of product reviews, spanning May 1996 -July 2014.


However, most if not all of the current automated metrics that are used in the NLG domain are based on overlap with a certain reference text.

it can be derived that such metrics are only useful if an aligned NLG output-reference text corpus can be constructed and if it can be assumed that qualitatively good output has a strong overlap with this standard.


PASS generates soccer match summaries aimed at fans of one of the teams that participated in the match.

the system can be seen as part of the 'Affective' NLG (ANLG) tradition, which aims to produce texts tailored towards the emotional aspects of the intended reader (Mahamood and Reiter, 2011;Ghosh et al., 2017, for instance).


Similarly, computer-generated texts were mistaken for human-written because they were vividly written, enthusiastically written and contained emotions.

the intended design of PASS, which was to produce texts that are similar to human in style seems to be successful according by these comments.


Texts that were tailored towards the preferences of the participant were neither seen as more aesthetically pleasing nor as more understandable.

no support was found for tailoring to have an effect on the attitude towards the text.


This difficulty level would allow researchers to evaluate a machine's NAQ detection performance more precisely.

we propose a method for automatically assigning difficulty level labels, which basically measures the similarity between a question and the target text passage.


However, the manual construction of such a dataset would be very costly.

this paper proposes an automatic dataset creation method for incorporating NAQs by making use of an existing MRC dataset.


These difficulty level annotations would allow us to choose a subset where the difficulty levels of NAQs could be adjusted as required.

we propose a method for automatically assigning difficulty level labels to NAQs.


If we only consider the results with answerable questions, the EM score was 0.607, which was moderately worse than 0.680, achieved with the model trained without NAQs.

the percentage   With the null-answer model, it is expected that correct answers will be recovered.


Moreover, the lower accuracy results confirmed that there is still scope for improvement in the NAQ detection models.

the created dataset could be effectively utilized in developing the NAQ detection functionality.


MultiTask-tag-style F I know that Dakota Fanning is only twelve, but she is really famous.

original F I would say that they do succeed but not frequently.


Tree-range fillers are represented by "..." enclosed as placeholders.


If we take out the entities and numeric values from the text, what is left can be considered some kind of templates with data slots waiting to be filled in.

the process of generating descriptions can be naturally divided into two stages: template generation and slot filling.


Given a sentence and a pair of an entity and a numeric value in the sentence, a relationship classification system can predict the relationship between the entity and the numeric value.

such a system can be used to assess the quality of automatic generations by extract correctly described records from the generated text.


Coster and Kauchak (2011a) imposed delete operation on their sentence compression model and improved the performance significantly.

deletion is also very important for abstractive sentence compression task.


Compression Ratio: The common assumption in compression research is that the system can make the determination of the optimal compression length.

compression ratios can vary drastically across systems.


In TSC, auxiliary memory takes a term as input and generates corresponding aspect representation.


These statistics suggest that passive or manual techniques for curbing propagation of abusive content (such as flagging) are neither effective nor easily scalable (Pavlopoulos et al., 2017).

the efforts to automate the detection and moderation of such content have been gaining popularity in natural language processing (NLP) (Waseem and Hovy, 2016;Wulczyn et al., 2017).


Therefore, the institutions undertaking this process are more likely to be educational technology companies with dedicated staff for assessment development, psychometric analysis, and natural language processing.

there is a business aspect to educational assessments in addition to research.


It is difficult to obtain coherent aspect embeddings if we only rely on the training of the sentiment classifier.

we add an unsupervised objective function to ensure the quality of the aspect embeddings, which is jointly trained with the attention-based LSTM.


(2017) where the annotations for the first set of the data contains the target span but the second set does not.

making it impossible to use the second set of annotation and forcing us to only use a subset of the dataset.


As the paper being reproduced does not define the number of epochs they trained for, we use early stopping.

for early stopping we require to split the training data into train and validation sets to know when to stop.


In other words, we can sample candidate c from a uniform distribution using the probability scores, s. However, we observe that the target rationales in the dataset often consist of few words -for example "room inside apartment".

we added the following Equation 4to limit the number of rationales selected, c, to be at most J.


Distant supervision exploits a simple assumption: if a sentence s contains an entity-relation pair (e1, relation, e2) that exists in a trustworthy knowledge source, then s can be a training data for the relation.

distant supervision often contains many noisy false-negative training samples and degrades the result.


From the experimental data, we find that the sentence representations in different languages are still far from each other and linearly separable.

it is hard for the multi-  Figure 1: Overall architecture of our adversarial multi-lingual neural relation extraction (AMNRE) which contains two languages.


Obviously, accurate relation classification results would benefit lots of natural language processing tasks, such as sentence interpretations, Q&A, knowledge graph construction, ontology learning, and so on.

lots of researchers have devoted to this research field.


This is because that the basic BRCNN model is based on SDP whose average word number is very small (about 4, Cai et al., 2016).

as long as a word is in SDP, it must be much important.


A cost of 1 is incurred if the system graph and the goldstandard graph disagree on whether a node (or edge) should be included.

the first component of the loss function max G (score(G) + cost(G; G * )) yields a decoded system graph that is slightly worse thanĜ = arg max G score(G); and the second component max G (score(G) − cost(G; G * )) yields a graph that is slightly better thanĜ.


There remains a significant gap between the current parser performance and the best it can achieve.

we notice that there is a gap of 1∼2 points in terms of ROUGE scores when comparing summaries produced using the two parsers.


We report limited length recall performance for both the metrics, as our system generated summaries are forced to be concise through some constraints (such as length limit constraint).

we consider using just the recall score since precision is of less concern in this scenario.


The matched entities in the database as well as the belief state for slots are concatenated as whole belief dialogue state, which is the input of dialogue policy.

the dialogue state b usually can be decomposed into some slot-dependent states and a slot-independent state, i.e.


However, in practice, for dialogue policy the probabilities of top K values of each slot play a more important role.

the whole belief state of each slot is often approximated by the probabilities of sorted top K values (Gašić and Young, 2014), which can be implemented by sorted K-max pooling (Kalchbrenner et al., 2014).


However, usually the graph is not known in practice, and the hypothetical structure is not guaranteed to be optimal.

it's better to optimize the structure along with the parameters of GNN.


Note that whereas the sampling still exists, the sampled noise is independent of the parameters φ.

the loss function L(θ, φ) is differential w.r.t.


One motivation of adding RNs is that RNNs tend to forget the distant history which RNs memorize it by explicitly introducing relations between pairs of words.

we assume that our method suppose to bring greater improvement on relative long sentences, which contains more distant history information than shorter ones that usually forgotten by RNNs.


As RNNs are not good at remembering the old history and cannot consider word relationship either, sometimes conventional NMT cannot get enough source information and hence emphasizes too much on the fluency of the target.

it suffers from meaning-drift and generates "inaccurate" translation.


SMT is a combination of several independent models; in comparison, NMT is an integral model itself.

most of these methods cannot be directly applied to NMT.


conduct a study that applys mixed fine tuning (Chu et al., 2017) on synthetic parallel data (Sennrich et al., 2016b), which shows better performance than either method.

we do not recommend any particular techniques in this paper but recommend readers to choose the best method for their own scenarios.


Even if out-of-domain data in the same language pair exists, it is possible that using both multilingual and multi-domain data can boost the translation performance.

we think that multilingual and multi-domain adaptation for NMT can be another future direction.


Transformer models have more advanced self-attention networks and more fine-grained multi-head attention mechanisms compared to RNN-based models with soft-attention.

transformer models have better performance in conventional translation tasks.


However, there are only very few unchanged translations in the MT task.

we explore a hybrid method, combining the NMT-based method and the dictionary-based method.


It also has the longest average token length.

nMT models can represent these larger vocabularies better than SMT models.


NMT models are powerful in using context information.

training the NMT models on sentence pairs is likely to improve the spelling normalization task further, which introduces more context information.


Then, we use the probability scores of the sentence as features.

each sentence will have n probability scores, one for each dialect.


Step 3: Relation annotation Annotators are given the same texts they annotated for emotions in Step 1 with the aggregation from Step 2 from all annotators.

all annotators see the same texts and annotations as input in this step.


This is feasible because characters, entities, and other were only annotated if they fill one of the roles, experiencer, target, or cause.

the prediction task boils down to a sequence prediction task of emotion phrases (for the different emotions) and the potential mentions of experiencers, targets, and causes.


However, this is something that our annotators do not have: all the context they have at their disposal is a triple of sentences, each of which can rely on information that is available in other parts of the book, but not in the annotation unit.

it is not always possible to annotate the cause, target, or even the experiencer.


Our experiments suggest that the prediction of emotions with their roles is a task that should be tackled with joint models.

this corpus adds an interesting relation extraction task to the set of existing challenges.


For example, say we would like to transduce the string say from its base form to its past form said.

the insertion context function might learn that there needs to be an insertion following the y that corresponds to the d. the insertion function values for say would be: and therefore the string we would try to transduce to said would be the string sayε, where ε is a special symbol marking potential insertion.


For the example above, the context we learn is yε$ where $ denotes the end of a string.

for a Figure 1: Graphical depictions of BiLSTMs (left), conditional random fields (middle) and refinement hidden Markov models (right).


To capture this, we also learn a pair of thresholds τ 0 and τ 1 (by grid search on the training set) such that Learning the Labeling Model with σ Once we have the σ function applied to all aligned training data, we mark in the output sequences in the training data the spurious insertions that should not be in the input sequence with a special symbol D (for deletion).

the learning algorithm is now also required to learn where to delete these extra insertions that σ elicits.


This is not particularly surprising given the fact that the Twitter dataset is composed of single words, each having a single typo.

each pair of misspelled and correctly spelled words often needs at most one insertion to be aligned.


Even when the translator's competency level in the source and target languages are the same, the target language could be influenced by the source language.

we trained all our models on language pairs whose target language is always English and recruited native English speakers for our evaluation tasks.


Similar to (dos Santos et al., 2014), our model is trained by contrasting positive and negative parallel sentences.

to compile our training set, given each source sentence, we generated a positive parallel pair by taking its correct target sentence and a negative parallel pair by taking a randomly chosen sentence from target sentences.


As we see in Table 3 there is a high correlation between the scores generated by the model and the BLEU scores of the parallel sentences.

we argue that irrespective of the language, the model is capable of extracting parallel sentences from any available language pair in Wikipedia with at least 95% accuracy in the last two σ regions.


Semantic shifts may be also be caused by changes in the attitudes of speakers or in the general environment of the speakers.

semantic shifts are naturally separated into two important classes: linguistic drifts (slow and regular changes in core meaning of words) and cultural shifts (culturally determined changes in associations of a given word).


It is rather straightforward to train separate word embedding models using time-specific corpora containing texts from several different time periods.


The reason is that most modern word embedding algorithms are inherently stochastic and the resulting embedding sets are invariant under rotation.

even when trained on the same data, separate learning runs will produce entirely different numerical vectors (though with roughly the same pairwise similarities between vectors for particular words).


This can be considered as an alternative to model alignment: instead of aligning models trained from scratch on different time periods, one starts with training a model on the diachronically first period, and then updates this same model with the data from the successive time periods, saving its state each time.

all the models are inherently related to each other, which, again, makes it possible to directly calculate cosine similarities between the same word in different time period models, or at least makes the models more comparable.


A famous example is the distributional model capturing the fact that the relation between 'man' and 'woman' is the same as between 'king' and 'queen' (by adding and subtracting the corresponding word vectors).

it is a natural development to investigate whether changes in semantic relationships across time can also be traced by looking at the diachronic development of distributional models.


Obviously, one main drawback of the above approach is that it much depends on the textual information and ignores the social link information which is important for user classification.

a better approach to learn user embedding is to incorporate both the textual and social link information.


The encoding of each source sentence is guided by the forest, and includes the forest nodes whose representations are computed in a bottom-up fashion using our ForestLSTM architecture ( §3).

in the encoding stage of this approach, different ways of constructing a phrase are taken into consideration along with the probability of rules in the corresponding trees.


The difference is that, in this model, the attention mechanism makes use of phrases as well as words.

the dynamic context is computed as follows: The TREE2SEQ model uses the top-1 parse tree generated by a parser.


Then, RAML optimizes log-likelihood on such sample given the corresponding input.

we need to sample auxiliary outputs from the exponentiated payoff distribution, q (y|x, y * ; τ ).


Therefore, it is common practice to compare parallel sentence extraction systems on artificially-created noisy parallel data by inserting non-parallel sentences into a parallel corpus.

to create our test sets we use parallel sentences from the newstest2012 and Europarl corpora and insert artificial noise by substituting target sentences with non-parallel target sentences from their respective held-out dataset.


Through our experiments, we observed that the optimal value ρ of BiRNN was constantly around a value of 0.99, while the one of the baseline system varied considerably from one test set to another.

the stability of our approach is preferable in practice.


The main challenge with our approach (and the majority of parallel sentence extraction systems) is that it needs to be trained on a parallel corpus.

we need parallel sentences to extract parallel sentences, which can be problematic.


If we take the $100 example, the canonical transition sequence is SHIFT, SHIFT, SHIFT, REDUCE S, SHIFT, REDUCE S, REDUCE I, and the corresponding cannonical parse is $ [ [ 1 0 ] 0 ] .

given w 0 , ..., w n−1 and P , the learning problem is mapping from w 0 , ...w n−1 to the canonical transition TRANSITION(P ).


It is clear that the deduction rules involve at most top two stack elements.

we extract the indices stored at the top of the stack which are potentially directly involved in any next action: l 1 , r 1 , l 0 , r 0 , and the buffer pointer i.


For the ITG RNN, the conditional probability is based on softmax over the output at each step t. As we mentioned in Section 2, for each transition sequence, there exists a functional mapping to a permutation P .

the probability of reordering is derived in the following way.


It reorders the encoder output of the input sequence before feeding it to the decoder RNN.

the burden of modeling reordering is shifted from the attention network to a dedicated reordering model.


Researchers and historians in the humanities and in the social sciences need to be able to query them and find answers rapidly.

new projects involve the domains of language processing, document recognition, and information retrieval for historical studies.


Moreover, an annotation step would be time-consuming for experts to create a ground-truth.

transductive transfer learning is an interesting approach when no ground-truth is provided.


results of this system showed that a language model was learned implicitly to generate the sequence of characters and that the correctly recognized words were words close from one language to the other.

without a specialized training on a small part of the data to be transcribed, the transfer learning did not work well.


In the scope of sequence generation, we padded the length of the sequence with blank labels up to 50 characters.

the network is free to define any sequence without any constraints.


The difference in languages implies a low coverage of the words but it still enables a correct decoding of the words as long as the resources come from the same time period.

the letter-ngrams models are efficient across languages sharing the same alphabet.


Particularly, we propose improved data collection methods through which we can obtain highquality large-scale datasets of labeled diagnosed conditions paired with appropriate control users.

we introduce SMHD (Self-reported Mental Health Diagnoses), a large dataset of diverse mental health conditions that can provide further insight into the mental health-related language.


In contrast to our problem, typical MT focuses on plain texts, and only unstructured parallel texts are normally available for training MT models.

we convert facts into natural language expressions beforehand.


(2017) proposed five different alignment models and reported the fourth variant performed best in their experiments.


Their annotations were aggregated by majority voting.


For example, a writer who created plenty of fake news is likely to create more fake news (Del Vicario et al., 2016); while news from authoritative organizations such as governments is less likely to be false.

the availability of multiple sources related to fake news has the great potential to help fake news detection.


We should emphasize that attention scores a i (1 ≤ i ≤ k) are learned along with other parts of the model in an end-to-end manner where they are optimized to capture the informativeness of the different sources at the fake news detection task.

capturing contributions of different sources reflected at scalar values a i (1 ≤ i ≤ k) makes the proposed framework somehow interpretable.


However, in practice, a piece of news might be a mixture of factual and false statements.

it is crucial to classify fake news into multiple classes reflecting degrees of fakeness.


In general, the two aforementioned goals can be regarded as optimizing the precision and recall of the clusters in the learned feature space.

we include two terms in MDF: one which pushes the centers by a margin referred as inter-class term and one which pulls the samples toward their centers referred as intra-class term.


As mentioned before, the original dataset (Wang, 2017) contains only the sources "S1: Statement", "S2: Metadata" and "S3: History" and we enrich the dataset by adding the fourth source "S4: Report".

we only report combinations with news content (i.e., S1), all sources in the original dataset (i.e., S1+S2+S3) and all sources in the enriched dataset (i.e., S1+S2+S3+S4).


In fakeness discrimination, we treated all classes the same.

we would like to incorporate class differences by enforcing larger margin between certain classes.


2017introduced sluice networks as a generalization and combination of various MTL architectures that use different information transfer techniques.

it can emulate, e.g., hard parameter sharing (Caruana, 1993), where the same layer weights (i.e., the same layers) are used for different tasks, and cross-stitch networks (Alonso and Plank, 2017), in which trainable parameters decide the amount of shared information between layers of separate networks.


16 instances were labeled correctly using MTL-all, but misclassified by MTL-en; and the same number vice versa.

16% of all 200 test instances were classified differently by the MTL setups.


While the previous work for gathering the tweets has been restricted to using twitter search APIs 2 , the free usage is limited to only a history of 7 days of tweets.

we use twitter scraper for no limits with respect to temporal history, as it scraps the web search results of the given query and returns tweets from them.


Both methods are applied to the five language families in our sample for which reasonably reliable expert trees were available, and we show that the results are similar.

support is found for also including results for the second method, which does not require expert trees.


As just mentioned, however, the real impact should come not from the mere number of items, but from the number of cognate classes they produce.

not unexpectedly, the correlation is stronger yet between the optimal number of cognate classes and the number of languages (R 2 = 0.923) and also significant (p = 0.0094).


Most of the terms (around 76%) occur in more than one community, although no innovation is present in all the subreddits in our dataset.

around 24% of innovations tracked occur in just one community -some of these are clearly topic-related, e.g., pkemon in /r/pokemon, while others are more general purpose abbreviations that, in principle, could appear in any community, such as txs (thanks, in /r/Android) or omgz (oh my god/gosh, in /r/subaru).


We observe that innovators are significantly more central than other users (for all measures considered: degree, betweenness and eigenvector) and are significantly more active in terms of number of posts than other individuals in the community (unpaired Welch's t-tests, all with p < 0.05).

a very robust pattern emerges across all subreddits, showing that innovators do have a particular profile in terms of their social standing: they do not belong to tightly connected cliques and occupy a central position in the network, as hubs with many connections of relatively low strength.


In order to reduce the number of labels, we replace the label with * for input tokens that are identical to their labels.

for the previous example, the labels are as follows: Ensembles of classifiers combine the results of multiple classifiers, in order to improve performance.


This is not possible for all languages because many do not contain a single table which would give forms corresponding to the full set of MSDs.

we limit our experiments to those languages where we can successfully find lexemes with a maximal number of MSDs.


For example, if the PAST and PRES;PCPLE forms in P are x 1 +ed and x 1 +ing respectively, we do not want to infer a form x 1 +x 2 +n for PAST;PCPLE because the stem corresponding to P has only one variable x 1 .

we restrict the vote to paradigms which have the same number of variables (x i s) as P .


Although there are examples of other PAST;PCPLE markers, the majority of words belong to one of these classes.

we would expect most form patterns in the past perfect slot to look similar to x 1 +ed or x 1 +x 2 +n.


This has the effect of preferring MSD realizations which are common.

sampling will indirectly minimize the number of distinct MSD realizations, which is our goal.


11 Completed entries v m may not be one-hot vectors.

we transform them into one-hot vectors by applying the transformation v m [arg max i v m [i]] = 1 and v m [i] = 0 otherwise.


To evaluate the method, we, after inferring the complete paradigms, substitute stem parts in to form patterns, for example x 1 +ed → eased.

we end up with a complete inflection table.


We estimate f (t|s, θ st ) as m i=1 f (x t i |s, θ st ), where x t i is again the i th coordinate of t. Each univariate density f (x t i |s, θ st ) is estimated using a Bayesian linear regression model that predicts the possible representations of the target word from s. However, this is only feasible if s has at most n − 2 coordinates.

we use a low-rank approximation of the source word representations, as follows.


However, word embedding models use only one vector to represent a word, and are problematic in some natural language processing applications that require sense level representation (e.g., word sense disambiguation, semantic relation identification, etc.).

some researches try to resolve the polysemy and homonymy issue and introduce sense level embedding, either act as pre-process (Iacobacci et al., 2015) or post-process  fashion.


Although the semantic relatedness datasets are used in many researches, one major disadvantage is that the words in those word pairs do not have contexts.

we also conduct experiments with the Stanford's Contextual Word Similarities (SCWS) dataset (Huang et al., 2012).


(2018) shows the attention mechanism is so powerful that removing other connections between the encoder and decoder has little effect on BLEU scores in machine translation.

a VED with deterministic attention might learn reconstruction mostly from attention, whereas the posterior of the latent space can fit to its prior in order to minimize the KL term.


But introducing measurement functions also allows more flexibility.

returning to our "lackluster" example, we can include a function that is one if that word is in the example and the label is negative: measurement features map the data into an extremely sparse and high-dimensional (partially) observed space.


This means that until the dataset is annotated once, the denominator of the accuracy calculation is growing.

inferred label accuracy reflects the corpus accuracy over the subset of documents that have at least one label.


It also tells us which Word-level model PTB WikiText-2 Small Medium Small Medium LSTM + ND (Jozefowicz et al., 2015) -79.8 --LSTM + ND (Kim et al., 2016) 97.6 85.4 116.8 † -LSTM + ND (Zaremba et al., 2014  Removing dropout of the context state: According to (1), context state s t changes linearly and thus should not suffer from over-fitting.

it seems reasonable to try to not regularize it and apply dropout only to the hidden states, i.e.


Also, notice that α = 0.99 brings the SCRN's results closer to the LSTM's results on non-English data.

we think LSTM stores too much of the long-term memory via its trainable forget gate, while in SCRN we can directly control this through the α.


In this formulation we use β t,i to capture the saliency of the dependency edge pointing to the i-th source word.

an edge w j ← w i has its salience score saved in β t,j ; and conversely, an edge w j → w i has its salience score in β t,i .


Furthermore, the situation of "the service generated an inappropriate expression by itself" is significantly worse than the situation of "a user posted an inappropriate question to the service, and the service displayed it".

it is difficult to directly use abstractive methods for commercial services from a business standpoint.


This task took nine days and was answered by 1,558 workers.

our dataset consists of 10,000 questions, each of which has more than five headline candidates with accompanying "headline-ness" scores.


Monte Carlo Rollout The disadvantage of REINFORCE algorithm in language generation is that the reward can only be assigned to a complete sentence.

every single action taken for generating a complete sentence share the same reward.


The intention of building such dataset is to teach model to response to the questions with respect to concepts in some certain images.

questions in VQA are much simpler than those ones in VQG.


Further analysis reveals that questions generated are not relevant to the target image.

the strategy of reusing questions from similar images is not sufficient for generating question with high quality.


In other words, the model has too many degrees of freedom.

words that are more general will not necessarily have a large variance.


However, for the particular semantic role and negation tasks described here, the focus is on identification of structural dependencies between words, which are not in this case sensitive to the specific nouns/verbs used.

for these tasks-as long as vocabulary words are not out-ofvocabulary for the models (which we confirm below)-the important thing should be not what the words themselves are, but whether dependencies between them have been captured in the sentence embeddings.


In this work, we investigate concatenation (concat), element-wise summation (sum) and element-wise multiplication (mul).

each document D and query Q is represented as R d×k matrix where d denotes the dimension of word embedding and k is the number of words in the input.


Different from the current cloze-style datasets for English reading comprehension, such as CBT, Daily Mail and CNN (Hermann et al., 2015), the three Chinese datasets do not provide candidate answers.

the model has to find the correct answer from the entire document.


If the rare words are not initialized properly, they would also bias the whole word representations.

a model without OOV mechanism will fail to precisely represent those inevitable OOV words from test sets.


Moreover, uncovering of a preposition is enough for some NLP application like Hindi-to-English translation (Kulkarni et al., 2012).

prepositional paraphrasing is a useful first step towards noun compound interpretation.


as encoders to learn their representations.

in this paper, we raise the following question: Can sequence representations, learned from word embeddings of components of a noun compound, help in its prepositional paraphrasing?


(2005) used true prepositions as an additional feature in their classifier for fine semantic relation prediction and reported significant improvement in their performance.

knowledge of preposition will definitely help in predicting and labelling finer relations.


Lauer (1995)'s dataset is not publicly available.

we extracted test samples from her thesis (Lauer, 1995).


(2005)'s dataset is not available online 2 .

we use Girju (2007)'s dataset (which has cross-lingual information along with English noun compounds) and extract relevant information to create a dataset.


Our objective is to learn representations such that similarity of a noun compound representation is higher with correct preposition compared with its similarity with any other prepositional paraphrase.

for each noun compound, we treat the remaining prepositions as negative examples.


Following this, a max-pooling operation is performed across the length of each feature map.

for all M feature maps computed from o is projected onto a dense layer with d p neurons followed by the final sigmoid-prediction layer with 5 classes denoting the five personality traits (Matthews et al., 2003).


During exploration, this is primarily observed for contextual comments which are very long.

sequential discourse modeling using RNNs may be better suited for such cases.


(2015) humour classifier as our anchor candidate scorer for simplicity but their HA extraction algorithm is able to work with any humour recognition model so long as it is robust to word order and capable of generating a humour score (in our case, we used humour probability).

using a more accurate humour model may have led to better performance.


We first find that FNC-1's proposed evaluation metric favors the majority class, which can be easily classified, and thus overestimates the true discriminative power of the methods.

we propose a new F1-based metric yielding a changed system ranking.


Nevertheless, the metric fails to take into account the highly imbalanced class distribution of the three related classes AGR, DSG, and DSC illustrated in Table 2.

models, which perform well on the majority class and poorly on the minority classes are favored.


In addition to the issues with the evaluation metric, there is also no upper bound reported for the FNC-1 data, although this will help estimating the headroom of the proposed systems with regard to human performance.

we ask five human raters to manually label 200 instances.


This is most likely because of the 100 new topics in the test dataset, which have not been seen during training.

the evaluation on the test set can be considered as an out-of-domain prediction.


In the error analysis, we observed that the feature-based systems lack semantic understanding.

we combine a feature-based system with a model that is better able to capture the semantics based  Table 4: Results of the feature ablation test.


We expect to measure the differences from the perspective of sentence complexity.

the number of noun phrases(N P count), verb phrases(V P count), prepositional  • Phrase Length Ratio: The length ratio(LR) of PP, NP, VP is computed respectively.


This method uses all features described in Section 2 except the KNN features.

all the used features are motivated by the humor theories.


This phenomenon can be explained by superiority theory (Keith-Spiegel, 1972;Gruner, 1997) that humor is the result of comparing with others.

it is unavoidable to mention people with pronouns.


This gives strong incentives for imposters to game the system.

fraudulent reviews flood the e-market websites.


In addition, four had questions answerable with common knowledge, while 29 had questions that were not based on one of the six question words and thus answerable with "yes" or "no."

we found that 93.6% of the QA pairs correctly conformed to the specification.


Afterwards, we replace the line-by-line decision layer with the poem-level encoder and final decision layer and train towards the per-poem decisions based on the parameters estimated before.

the final model is able to steer its attention mechanism towards the important lines and can learn to sacrifice the initially trained per-line optimization for the overall per-poem optimization.


Cell Scoring: In most cases, the answer candidate cells in each row are not components of a question.

for each (q, T (r, c)), input of the encoder does not contain the candidate answer cell T (r, c), and it is replaced by a tag " SPACE ".


Besides, our model runs faster than LSTM, and has almost the same running time with CNN for each batch of data.

diSAN works well on this task in our model.


Bag-of-words: BOW is a simple vector representation of an entire document where each dimension is representative of a unique word in the corpus.

the input dimensionality is equivalent to the number of terms which are maintained when building the BOW representation.


16 With sufficient labeled data, the neural transfer approaches tend to do well, but of course, these correspond to cases with less data scarcity.

we encourage researchers to compare neural network approaches with simpler baselines such as PRED and PRED-CCA, rather than only comparing against the "no tranfer" option, and to report not only F1 scores, but also precision and recall, which may be useful when deciding which transfer learning approach to use.


A diagram showing the turbo training algorithm is given in Figure 3.

it can be proved that the three  Figure 3: Turbo training steps in the turbo phase correspond precisely to coordinate-descent minimization of L. iterations in the turbo phase provably decreases the loss L monotonically and are guaranteed to converge.


The contributions of this paper can be summarized into two parts: 1) We propose a novel SNN to extract semantic and structure representations, which enriches information of sentence representation.

sNN improves the performance of text classification.


In the first sentence, the structure "what is ... the name of" appears with almost the same frequency (67 and 66 times) in human and entity class and appears very little in the other classes.

more weights should be added on semantic representation.


It shows a strongly positive correlation apart from a few small random variations caused by a limited sample size.

it proves that our attention mechanism is very reasonable.


As the accuracy of lyrics segmentation in state of the art is not fully satisfying yet (Watanabe et al., 2016), and given the variability in the set of structure types provided in the literature according to different genres (Tagg, 1982;Brackett, 1995), rare attempts have been made to achieve semantic labeling.

in our work, we focus on improving the state of the art in lyrics segmentation, leaving the task of semantic labeling for future work.


Even when segment borders are "indicated" by line breaks in lyrics available online, those line breaks have usually been annotated by users and neither are they necessarily identical to those intended by the songwriter, nor do users in general agree on where to put them.

a method to automatically segment unsegmented song texts is needed to automate that first step.


We employ the same motivation here: segment borders manifest themselves in the form of distinct patterns in the SSM.

convolutions allow the network to capture those patterns efficiently regardless of their location and relative size.


In CNN models, the information lies in the feature maps.

the biggest difference of the multi-task architecture is the method for fusing the feature maps.


The problem with this dataset, however, is that answer triples are provided from Freebase, which has been defunct for several years.

it is difficult to build "real-world" question answering systems that are operationally deployable.


Second, a defunct knowledge graph means that researchers must develop custom infrastructure for querying, browsing, and manipulating the graph.

we are not able to leverage multiple cooperative and interchangeable service APIs that are deployed and maintained by different parties-which is the strength of the broader "open linked data" ecosystem.


It can be doubted that emotions are expressed in a comparable way in these different domains: Journalists ideally tend to be objective when writing articles, authors of microblog posts need to focus on brevity, and one might assume that emotion expressions in tales are more subtle and implicit than, for instance, in blogs.

the transfer across emotion recognition models is, presumably, challenging.


In most datasets, each instance is only annotated discretely with single labels (exceptions are SSEC and AffectiveText).

most instances have exactly one label marked with a 1.0.


Creation of lexicons-documents graph: Since each label specific lexicon term can occur in multiple documents, and one document can contain many such lexicons, in order to create a disjoint partition of lexicons and documents into training/test sets, it is natural to represent documents and lexicons using a graph-based structure.

for each class, we create an undirected graph whose nodes are lexicons and documents respectively.


(Sequiera et al., 2017) While previous work has recognized the increasing use of deep learning techniques in natural language processing (Young et al., 2017), no systematic survey of deep learning methods for answer selection to date has been published.

in this paper, we aim to give a comprehensive review of various deep learning methods that have been used to tackle the answer selection task.


For example, while many Siamese architectures typically capture less interactive features between the input sentences than the Attentive architectures, few recently proposed Siamese architectures have sophisticated comparison layer after the encoding layer (He et al., 2015;Rao et al., 2016).

they even outperform some Attentive architectures.


For example, in a cQA platform, it is not unusual for a question to have hundreds of answers, the vast majority of which would not satisfy a user's information needs.

using answer selection techniques to find relevant answers can be very beneficial.


NER systems are often used as the first step in question answering, information retrieval, co-reference resolution, topic modeling, etc.

it is important to highlight recent advances in named entity recognition, especially recent neural NER architectures which have achieved state of the art performance with minimal feature engineering.


For example, "皮鞋(leather shoes)" is included in the dictionary, while "皮带(leather belt)" is not included.

in Figure 1(b), "皮鞋(leather shoes)" is annotated as a PDT, but "皮带(leather belt)" is not annotated.


It is a common problem known as false negative instance which may misguide model if we arbitrarily label them as "O".

we consider that each non-matched character can be annotated as any proper label.


We expand the original model for this situation and apply softmax over all candidate output label sequences, thus the probability of a distantly supervised instance is computed as follows: x,ỹ) ȳ∈Yx e score(x,ȳ) We exploit a negative log-likelihood objective as the loss function.

the loss function of our model with CRF-PA can be computed as follows:  Figure 3: An example instance which is partially annotated.


In particular, if the sentence is annotated by hand and each character has definite label, the set z includes only one label sequence.

the above objective function is also suitable for supervised instances.


In these tables correct/incorrect mean whether the local prediction is correct or not (comparing to the ground truth).

the first cell with value 1134 indicates that there are 1134 mentions in test-b that are correctly predicted by the local model, but deemed as among the top 25% least confident mentions (aka the hard queries) by h 1 .


From our preliminary experiments, we found out that keeping the transferred parameters fixed during the subsequent step usually produces worse results for both models.

in the subsequent step, the transferred parameters are not fixed but updated together with the rest of the network.


This may be explained by the fact that the MISC category is a collection of NEs that do not fit into other categories.

the context of such NEs may vary a lot from one to another, making the recovery from the previously-learned adverse knowledge on this NE category more challenging, e.g., its new context and information in the subsequent step are sparser.


sources, namely, question-answer collections and review collections contain valuable information for generating instant responses for product questions.

given a question, our framework can return a ranked list of snippets serving as the automated response, where each snippet can be a sentence from reviews or an existing question-answer pair.


The following snippets come from reviews and they mention that the cable is not for data transferring.

the user can receive informative response extracted from both the review collection and the QA collection associated with the corresponding product.


The splitting of documents and removal of disfluencies result in different sentence and token identifiers, since sentence numbering restarts for each document part and token count excludes disfluent tokens.

the CoNLL-2011 release is required for resolving the identifiers used in the positive interpretations dataset.


Unfortunately, due to time constraints, the authors were not (yet) able to provide us with the train/test splits as used in their paper.

we created our own test set using the same approach: we used a 80/20 split and made sure that all positive interpretations belonging to one negated verb were assigned to either the train or the test set.


However, intuitively, it is clear that the scores 2 and 3 are somewhat problematic in the sense that they present the middle cases that less clearly belong to either class.

these two scores could potentially make up their own class of UNCERTAIN.


The ARG1, the advance, is coreferential with gained in the preceding context; the new, relevant information is the lack of participation of many issues in this advance.

in this case, the positive interpretation generated from this semantic role is FALSE instead of the more frequent TRUE class associated with ARG1.


If the sentence does contain focus, negation yields a focus-associated reading (6a) and a non-associated reading (6b), as Kawamura (2007) illustrates with the example below.

this ambiguity has been regarded as a result of the scope interaction of focus and negation (Jackendoff, 1972;Krifka, 2006 we think  improved upon the task by scoring each and every potential positive interpretation generated from the negation.


relation labels and facts (Fan et al., 2017).

unlabeled entity pairs are likely to be wrongly forced into existing relation labels by distant supervision approaches, instead of their true, unknown labels.


We can see that the DSNN only takes the training data of known relations directly derived from a domain knowledge graph without requiring any training data for new relations.

our approach is a very weakly supervised one with minimal human intervention.


Entities "张智霖 (Zhilin Zhang)" and "袁咏仪 (Yongyi Yuan)" are in the coordinate dependency relation via a direct arc.

the SDP between two entities is "张智霖 (Zhilin Zhang) → 袁咏仪 (Yongyi Yuan)", which contains not much useful information.


The reason that goodness-of-fit models fail is that they are sensitive to check whether the data follows uniform distribution or not, while our purpose is to select those with prominent peaks.

non-confident relation labels are assigned to unlabeled entity pairs roughly, increasing the number of false positive instances.


Although all of these sentences mention the names of interests, some of them might not express police killing incidents.

it is crucial for the systems to be able to model the multiple sentences in the containers appropriately so that the correct sentences for police killing events can be captured to perform classification for the sentence containers of names.


Other examples aggregate information across conjuncts as in the sentence, "The average of 3 and 5 is 4."

we create a list of triggers ('between', 'among', 'total', 'sum', 'average','each other', ...) using Thesaurus expansion of seed words which indicate nondistributive coordination .


First, relations often span over long nested structures or are presented in a non-canonical form that cannot be easily captured by a small set of extraction patterns.

such relations are commonly missed by state-of-the-art approaches.


paraphrasing) for specific syntactic environments in which they occur.

the transformation process is carried out in a top-down fashion, starting with the input document and using a set of syntactic rule patterns that define how to split up, transform and recurse on complex syntactic sentence patterns (see Section 3.1.3).


However, the general quality of the verb representations is much lower than the quality reported for nouns.

the mapping to unseen verb pairs yields unsatisfactory results for the full SimVerb dataset.


In NLI this arises when the test set contains many easy examples, and the extent to which difficult components of language understanding are required is masked within traditional evaluation.

we ask a natural question: is good model performance on NLI benchmarks a result of sophisticated pattern matching, or does it reflect true competence at natural language understanding?


Our constructions also lead to high word overlap for contradiction pairs, leading to a large number of C-E errors for both models (1695 out of 4337 errors).

78.3% of all errors are caused due to the models falsely predicting entailment.


Evaluating the ability to infer information from a text is a crucial test of the capability of models to grasp meaning.

the computational linguistics community has invested huge efforts into developing textual entailment (TE) datasets.


1-4 as a single GRU step as: we interpret each GRU step as a composition function of two objects: h i−1 and x i .


We hope this start point guides where the path goes and what state to reach at each position.

relations lie in the input space and entities lie in the hidden space.


Figure 3 shows that we extend the basic ROP architecture ( §4) by a GRU layer (bottom layer in Figure 3), denoted as GRU r , that learns the representation of the relation sequence.

finally, we use h t (the last hidden state of GRU r , shown in blue in figure 3) to match the query relation r. 3 the training loss of this task has two parts: one comes from the loss of ROP training (l seq , Eq 18); the other is the prediction loss of query relation, denoted as l pred .


Our ROP models achieve this implicitly, as the relation sequence can predict the entity sequence; this makes sure the representation of the relation sequence is specific to the entity sequence.

the same goal can be achieved as (Das et al., 2017).


ROP predicts intermediate entities and updates the entity/relation embeddings and other parameters in each step; the training signals can come from the in-path target entities and from the reasoning task specific annotations.

rOP makes use of the in-path structures to learn good quality entity/relation embeddings, which are further employed and finetuned to solve the reasoning problem.


Spearman correlation coefficients between these percentages and system performance are always higher than 0.9.

the more inverse relations, the worse performance.


Instead, its incorrectness cannot be determined without understanding the meaning of the whole sentence.

besides directly adopting the techniques used for English GEC, many aspects of Chinese GEC are worth studying.


(E3-3) 可是每個人的(*對應, 反應)都千差萬別 ( but everyone's (correspondence, reaction) is different ) Given the original segment, one can conclude that the candidate "境況" (situation) selected by Con-text2vec is less suitable compared to the ground-truth "反應" (reaction), according to the similarity criterion.

wUE correction is different from sentence completion in that if the model ignores the word originally written by the language learner, it is likely to generate a correction that changes the meaning of the original sentence segment.


Though we can let the cutoff be a function of various factors, it is difficult to design such a function explicitly.

instead of performing combination of scores, we combine the rank proposed by the LM with the rank based on our correction generation model.


Though sharing a common Chinese character, the meaning of "一起" (together) and "一直" (always, all the time) are not quite similar.


This example shows that the single-answer automatic evaluation can underestimate the performance of our system.

we perform human evaluation, in which the top candidates proposed by our model are judged by annotators.


Each annotation instance is assigned to two annotators randomly, so we can assume that the difficulty of the data assigned to each annotator is the same.

we calculate the average inter-annotator agreement of all pairs of annotators.


In context-sensitive generation of open-domain conversational responses, the input message to the encoder usually includes several historical utterances in a conversation.

one of the key problems in context-sensitive generation is how to encode historical utterances in a conversation.


The segmentation process reduces the vocabulary and alleviates the data sparse problem.

the learned word embedding for sub-word is more robust, then the performance of the PB prediction system can be improved.


Synonymy, as viewed by (Lyons, 1968;Cruse, 1986), has to be seen as a scale of similarity (absolute, near and partial synonymy) and it is generally acknowledged that absolute synonymy is rare in natural languages.

it is believed that context must be taken into account (Palmer, 1981).


Let us assume that the relation r is represented by a vector r ∈ R δ in some δ-dimensional space.

we can write r(h, t) as a function that takes two vectors (corresponding to the embeddings of the two words) as the input and returns a single vector (representing the relation between the two words) as given in (1).


If δ < d, we can first project word embeddings to a lower δ-dimensional space using some dimensionality reduction method such as SVD, whereas if δ >d we can learn higher δ-dimensional overcomplete word representations  from the original d-dimensional word embeddings.

we will limit our theoretical analysis to the δ = d case for ease of description.


However, such a trivial solution would not generalise to unseen test data.

in addition to D + we would require a set of non-analogous word-pairs D − as negative examples.


Expected correlation of different dimensions in the same word is zero from (8).

such cross-correlations are likely to be small for different word pairs, which are nonsynomyous.


If we think of H and T as d 2 -dimensional word embeddings, E p + [H il T jm ] represents the expectation over two distinct dimensions of H and T for il = jm.

from the same logic as above, this expectation is approximately zero.


A similar argument can be used to show that terms that involve A ij Q kn disappear from (10).

a does not play any part in the expected loss over positive examples.


Similarly, we can show that A is independent of the expected loss over negative examples.

from (5) we see that the expected loss over the entire training dataset is independent of A.


(2017) hypothesize that this phenomenon correlates with shifting, which can be seen as producing a new (negative) end state.

they collect particle verbs containing relevant English particles, such as away, down and out.


Endangered languages pose additional problems, as the lack of resources is further exacerbated by the lack of standard orthography.

there is an obvious need for both resources and technologies adapted to languages of under-represented communities.


In addition, we find that the word segmentation in our collected narratives follows more the concept of a phonological word.

words that are segmented in the UoI corpus, in our narratives are often fused in a single token.


A direct result of this phenomenon is that annotating such tokens with single POS tags does not capture all of the necessary information.

we chose to annotate such words with multiple POS-tags, effectively making our tag dictionary the superset of the universal tagset.


We concatenate these character representations to constitute a word, which is represented as r ∈ R |Vc|×n where V c is a vocabulary of characters and n is the length of a word.

the input representation for convolution has the following shape: where c i is a one-hot encoding for representing the i-th character in a word and ⊕ is a concatenation operator.


To extract salient features, we apply a max-pooling operation on the vector s. Specifically, we use max pooling with stride to derive locally salient features that can contain suffixes, roots, and prefixes.

each feature derived from stride pooling has a summary of subword information in a character sequence.


The equation for stride pooling is as follows: where k is the length of a stride and s k•i:k•(i+1)−1 is a sequence representation with one stride in s. As stride pooling results in features that vary in length according to the length of words, we therefore simply sum the elements of the vector e to produce fixed-sized features.

far, we have described the procedure for a single filter.


The error in this case can be avoided by using the Zero Width Non-Joiner (ZWNJ) 4 Unicode character but Urdu users are generally not aware of its existence and most Urdu keyboards also do not have a direct key mapping for this character.

an extra space within a word creates two (or more) separate tokens for a single word and creates a computational problem.


Our semantic signatures based on associated ideas cover all synonyms and senses of the test set.


More interestingly, the model trained on the Manulex-3N dataset (using the simple rule of first occurrence) clearly outperforms the model based on Manulex-Cont.

we decided to retain the model based on the first way of defining φ(D) for the final version of ReSyf.


VMWEs are known to have specific lexical and morpho-syntactic variability profiles: they are more or less variable, but usually not as variable as regular phrases with the same syntactic structure.

methods used for detecting paraphrases of regular phrases (Fujita and Isabelle, 2015) cannot be straightforwardly applied to VMWEs.


Note that this approach neutralizes both morphological variation (via lemmatization) and syntactic variation (we disregard the labels and directions of the dependencies).

the predicted STVs can be either true positives (Fig.1.a and 1.b) or false positives ( Fig.1.d).


When considered as a sequence, we sort lemmas lexicographically to neutralize variable word order, as in (6) vs. 9, and obtain the Normalized Form (NF).

4 le;page;tourner 'the;page;turn' is the ABS NF of the VMWEs in (6)-(10).


Finally, the NFs for all candidates are produced (Norm).

of the previous phase, we obtain sets of candidates grouped by their NFs (cf.


2 However, studies show that its archaic themes and small corpus size (1,189 chapters) can limit performance (Hao et al., 2018;Moritz and Büchler, 2017).

the requirement of parallel/comparable corpora for multilingual topic models limits their usage in many situations.


In this model, the topic assignments for both documents are independently generated from the same distribution, θ d 1 ,d 2 .

the joint likelihood of document links model is: where w d and z d are the word tokens and topic assignments of document d .


Obtaining parallel corpora for training the document links model is very demanding, particularly for low-resource languages.

as the second major contribution in this paper, we generalize the document links model using the formulation described above to require only a bilingual dictionary.


Intuitively, if d 1 and d 2 have a large amount of overlapping word translations, their topics should be similar as well.

we define the values of δ based on the similarity of document pairs using a bilingual dictionary.


They argue that many abbreviations in the biomedical domain follow a predictable pattern where the first letter of each word is a letter of its short form, but that there are cases where this is not true.

they describe a simple and fast algorithm with a high recall and precision to find those pairs using string operations to verify that the acronym letters appear in the right order in the long form.


(2015) use the sum of all pmi (pointwise mutual information) values between a definition and a context as a measure of their similarity.

they also use a more advanced similarity measure than just word overlap based on word co-occurrence in a large corpus.


Our data set is generated automatically and not manually corrected.

we have some errors in our data set that cause misclassifications.


This way we can still classify over 76% of all examples.

we can resolve a large number of acronyms in our database.


Further, Meinhof's 1948 noun class tables of singular/plural pairs is underspecified as to which option is used when.

we have revised the tables so as to achieve a deterministic selection during pluralization.


Byamugisha et al., (2016) handled this by placing an 'm' on the NC, while prefix exceptions were directly associated with their plurals.

we investigated revising Meinhof's NC definition for other agglutinating Bantu languages within and across Guthrie zones.


We extracted it as a part by ignoring the negation.

the results of the annotation were illuminating: helping to pinpoint where SUMO is too general for our purposes, or where our extraction script needs refinement.


However, MilitaryVehicle has Vehicle as its parent and inherits from it.

the telic quale for Vehicle, which is extracted as "Translocation," would be inherited by MilitaryVehicle.


Although our intention has never been to create a detailed ontology of the entities that serve as the participants in these events/states, such information is needed to generalize and map this information across resources, as well as provide some insights into selectional restrictions generally and qualia relations specifically.

we opted to integrate REO with an existing ontology containing the type of information on participants that we were interested in.


FEs have a communicative function and thus in FE dictionaries, they are classified into categories on the basis of their functions.

if a user wants to write about the fact that there is little research on a topic, the user goes to the category 'to show lack of existing research' and finds expressions such as 'there has been little discussion about', even if the user has not ever encountered the entire expression.


Despite the usefulness of such FE dictionaries, it is time-consuming for users to choose a category and manually find candidate expressions every time they consult the dictionary.

the aim of our research is to enable semantic searches of FE dictionaries by computers with incomplete user input.


FEs recorded in the dictionaries are not always suitable for writing assistance systems because many of the expressions do not occur in an actual corpus and differ depending on disciplines.

we aim to build a more comprehensive and more domain-specific FE dictionary by expanding an existing FE dictionary.


However, these studies specifically addressed characteristic phrases that were informative.

fEs such as 'in this paper' were not considered target expressions.


3 Semantic FE Search for Writing Assistance System Most existing writing assistance systems seek candidate expressions using keyword-based searches, which causes the problem wherein expressions very different from the input are not presented to users.

we first consider a semantic searching system that searches for FEs.


Then, the user picks one of the FEs belonging to that category.

the proposed searching process consists of two steps.


To choose the correct category, the system must use information related to context that is derived from what a user is writing.

a resource must include information related to context in which an expression is used.


However, specifically examining scholarly papers shows that the structure of scientific documents is fixed to some degree.

as long as users write along this fixed structure, there is a good chance that the system can anticipate what the user wants to write if the categories correspond to the paper structure.


Consequently, as long as users write along this fixed structure, there is a good chance that the system can anticipate what the user wants to write if the categories correspond to the paper structure.

on the basis of the logical structure, categories correspond to both the functions of FEs and the user's intention.


These might appear to be useful, but they cannot be assigned one category label because 'on the other hand' can be used in many sections of a paper.

we chose to exclude them from this research.


This is most problematic when using the existing dictionary because we cannot search corpora for sentences that contain FEs, using only formulaic parts.

for formulaicity to be learned from the dictionary, we first annotated the formulaic parts in the dictionary.


Annotation guidelines are available on our GitHub repository 1 .

category: Stating the purpose of research Formulaic expression: the objective of this research be to Example sentences from AcL Anthology: one of the objectives of this research is to make online documents more understandable by paraphrasing unknown … the objective of this research is to experiment with these techniques for Sinhala-Tamil, and identify the best … Figure 3: In the annotated dictionary, each FE is classified into a category and has several example sentences extracted from a corpus 3 .


User input can be an incomplete sentence.

inputs of two types are prepared as presumed user inputs.


These approaches have performed WSD by combining contextual information and semantic knowledge on the LKBs.

there is much number of words that can be disambiguated when compared to the supervised approaches.


The fixed routing policy cannot avoid this issue.

we wish for a new way to aggregate information according to the state of the final encoding.


The coefficients c ij is computed using a softmax function, and M j=1 c ij = 1.

the total amount of information transferred from capsule h i is proportionally summed to one.


When an output capsule v j receives the incoming messages, its state will be updated and the coefficient c ij is also re-computed for each input capsule.

we iteratively refine the route of information passing, towards an instance dependent and context aware encoding of a sequence.


In reversed DR-AGG, each output capsule is used as query vector to assign a proportion weight to each input capsule and then receives messages from the input capsules in proportion.

for all output capsules the total amount of message received by an output capsule is same.


By introducing the None label and the <VR> node, we construct a semantic tree rooted at the <VR> node with several virtual arcs labeled with None.

the predicate disambiguation and argument identification/labeling tasks can be naturally regarded as the labeling process over all the word pairs.


In SRL task, the distribution of the role labels is similarly uneven and the problem comes worse after we introduce the additional <VR> node and None label, directly applying the primitive form of bilinear attention would fail to capture the prior probability P (y i = c k ) for each class.

the biaffine attention introduced in our model would be extremely helpful for semantic role prediction.


First, since our goal is to predict the label for each pair of h , the output of our biaffine transformation should be a vector of dimensionality N r instead of a real value, where N r is the number of all the candidate semantic labels.


Though CoNLL-2009 provided the gold predicate beforehand, the predicate identification subtask is still indispensable for a real world SRL task.

we further augment our model with the predicate identification ability.


If two or more authors have very similar scaled scores (say 1.0 and 0.98) then they are equally likely to be the authors of the document (as per the feature type under consideration).

as the next step of this process we form groups of authors from the perspective of this feature type.


This is because a larger z-score cutoff means that the winning author must have a significantly higher plausibility value compared to all the other authors.

an author is announced for a document only if he/she is the clear winner compared to all the others.


In recent years, a significant amount of interest has built up around slot filling and intent detection due in part to the competition among commercial artificial intelligence assistants, such as Samsung Bixby, Apple Siri, Google Assistant, Microsoft Cortana, and Amazon Alexa.

a multitude of studies regarding recurrent neural network (RNN) models for slot filling and intent detection has been completed and has resulted in outcomes that generally outperform most other past approaches.


'Open-vocabulary' slots signify a slot type that can take on values from a virtually unlimited set, such as file name, album name, text body, or schedule title.

words that are not seen in the training set or are employed differently from the meaning inherent in them can be included in sentences because of the fact that they have no constraints on the length and specific patterns of content.


It configures positional encodings for each word index according to time step distance | − | from the current word index multiplied by a parameter .

it has fewer position-related parameters than the positionwise encoding.


The LM was trained by randomly removing a word in a sentence, inserting other words within window 10 (±5) of the removed word as an input, and using the removed word as an output.

positional encoding in the LM treats the index of the removed word as the current word index b = 5 in (14), and its 10 indexes of input are 0, … ,4,6, … ,10.


A final cost function to be maximized is as follows: (23) where λ is a weight-decay hyper-parameter for the LM and l2-norm, is the length of the input sequence, is the number of intents, and is all the parameters learned from training data.

the LM is used for stability and utility of character-based word representation and can be construed as a regularizer for the SFM and IDM.


7 SNLI has 3 classes of sentence-pair judgments: neutral, contradiction, entailment This module finds the closest sentence in the source document(s) with respect to each of the target sentences.

each sentence (t k ) in a target document (T ) is mapped to its nearest source sentence (s ij ) where i denotes the source document and j signifies the sentence position within that document.


We define closeness as the maximum of cosine similarity between the vectors a k and b ij .

bij | we have a mapping relationship for each sentence in the target document with one of the source sentences.


The first heuristic follows the most standard procedure of the Siamese architecture, while the latter two are certain measures of "similarity" or "closeness".

the Relative Sentence Vector (RSV) corresponding to a target sentence t k is represented as : where comma (,) refers to the column vector concatenation.


In our preliminary experiments, we find that a news document is more likely to be relevant to a firm only if it is mentioned in the news abstract.

we only include news documents mentioned at least one public listed firm in the U.S. security market.