query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : In natural language processing tasks, it is essential to accurately represent the meanings of words to achieve optimal performance. However, this task is challenging due to the potential for multiple interpretations of a single word. As a solution, I propose a method that initially learns to identify the smallest unambiguous phrase component within sentences and paragraphs. Following this, I aim to accurately model their latent vectorized representations, taking into account their semantic and structural relationships with neighboring semantic components. My current approach involves capturing the semantic relationship using a pretraining method akin to word2vec. To encapsulate the structural syntactic relationships, I propose modeling each semantic component as a node, with the entire paragraph represented as a sparse graph structure. Subsequently, specialized graph-based neural network algorithms can be applied to obtain latent structural representations. Ultimately, my proposed embedding model should assign a more precise embedding to each text input, potentially enhancing performance in numerous downstream tasks.

positive:
Existing neural semantic parsers mainly utilize a sequence encoder, i.e., a
sequential LSTM, to extract word order features while neglecting other valuable
syntactic information such as dependency graph or constituent trees. In this
paper, we first propose to use the \textit{syntactic graph} to represent three
types of syntactic information, i.e., word order, dependency and constituency
features. We further employ a graph-to-sequence model to encode the syntactic
graph and decode a logical form. Experimental results on benchmark datasets
show that our model is comparable to the state-of-the-art on Jobs640, ATIS and
Geo880. Experimental results on adversarial examples demonstrate the robustness
of the model is also improved by encoding more syntactic information.

positive:
Long text brings a big challenge to semantic matching due to their
complicated semantic and syntactic structures. To tackle the challenge, we
consider using prior knowledge to help identify useful information and filter
out noise to matching in long text. To this end, we propose a knowledge
enhanced hybrid neural network (KEHNN). The model fuses prior knowledge into
word representations by knowledge gates and establishes three matching channels
with words, sequential structures of sentences given by Gated Recurrent Units
(GRU), and knowledge enhanced representations. The three channels are processed
by a convolutional neural network to generate high level features for matching,
and the features are synthesized as a matching score by a multilayer
perceptron. The model extends the existing methods by conducting matching on
words, local structures of sentences, and global context of sentences.
Evaluation results from extensive experiments on public data sets for question
answering and conversation show that KEHNN can significantly outperform
the-state-of-the-art matching models and particularly improve the performance
on pairs with long text.

positive:
Most unsupervised NLP models represent each word with a single point or
single region in semantic space, while the existing multi-sense word embeddings
cannot represent longer word sequences like phrases or sentences. We propose a
novel embedding method for a text sequence (a phrase or a sentence) where each
sequence is represented by a distinct set of multi-mode codebook embeddings to
capture different semantic facets of its meaning. The codebook embeddings can
be viewed as the cluster centers which summarize the distribution of possibly
co-occurring words in a pre-trained word embedding space. We introduce an
end-to-end trainable neural model that directly predicts the set of cluster
centers from the input text sequence during test time. Our experiments show
that the per-sentence codebook embeddings significantly improve the
performances in unsupervised sentence similarity and extractive summarization
benchmarks. In phrase similarity experiments, we discover that the multi-facet
embeddings provide an interpretable semantic representation but do not
outperform the single-facet baseline.

positive:
Variational auto-encoders (VAEs) are widely used in natural language
generation due to the regularization of the latent space. However, generating
sentences from the continuous latent space does not explicitly model the
syntactic information. In this paper, we propose to generate sentences from
disentangled syntactic and semantic spaces. Our proposed method explicitly
models syntactic information in the VAE's latent space by using the linearized
tree sequence, leading to better performance of language generation.
Additionally, the advantage of sampling in the disentangled syntactic and
semantic latent spaces enables us to perform novel applications, such as the
unsupervised paraphrase generation and syntax-transfer generation. Experimental
results show that our proposed model achieves similar or better performance in
various tasks, compared with state-of-the-art related work.

positive:
Syntactic structure of sentences in a document substantially informs about
its authorial writing style. Sentence representation learning has been widely
explored in recent years and it has been shown that it improves the
generalization of different downstream tasks across many domains. Even though
utilizing probing methods in several studies suggests that these learned
contextual representations implicitly encode some amount of syntax, explicit
syntactic information further improves the performance of deep neural models in
the domain of authorship attribution. These observations have motivated us to
investigate the explicit representation learning of syntactic structure of
sentences. In this paper, we propose a self-supervised framework for learning
structural representations of sentences. The self-supervised network contains
two components; a lexical sub-network and a syntactic sub-network which take
the sequence of words and their corresponding structural labels as the input,
respectively. Due to the n-to-1 mapping of words to their structural labels,
each word will be embedded into a vector representation which mainly carries
structural information. We evaluate the learned structural representations of
sentences using different probing tasks, and subsequently utilize them in the
authorship attribution task. Our experimental results indicate that the
structural embeddings significantly improve the classification tasks when
concatenated with the existing pre-trained word embeddings.

positive:
Vector averaging remains one of the most popular sentence embedding methods
in spite of its obvious disregard for syntactic structure. While more complex
sequential or convolutional networks potentially yield superior classification
performance, the improvements in classification accuracy are typically mediocre
compared to the simple vector averaging. As an efficient alternative, we
propose the use of discrete cosine transform (DCT) to compress word sequences
in an order-preserving manner. The lower order DCT coefficients represent the
overall feature patterns in sentences, which results in suitable embeddings for
tasks that could benefit from syntactic features. Our results in semantic
probing tasks demonstrate that DCT embeddings indeed preserve more syntactic
information compared with vector averaging. With practically equivalent
complexity, the model yields better overall performance in downstream
classification tasks that correlate with syntactic features, which illustrates
the capacity of DCT to preserve word order information.

positive:
Capturing the meaning of sentences has long been a challenging task. Current
models tend to apply linear combinations of word features to conduct semantic
composition for bigger-granularity units e.g. phrases, sentences, and
documents. However, the semantic linearity does not always hold in human
language. For instance, the meaning of the phrase `ivory tower' can not be
deduced by linearly combining the meanings of `ivory' and `tower'. To address
this issue, we propose a new framework that models different levels of semantic
units (e.g. sememe, word, sentence, and semantic abstraction) on a single
\textit{Semantic Hilbert Space}, which naturally admits a non-linear semantic
composition by means of a complex-valued vector word representation. An
end-to-end neural network~\footnote{https://github.com/wabyking/qnn} is
proposed to implement the framework in the text classification task, and
evaluation results on six benchmarking text classification datasets demonstrate
the effectiveness, robustness and self-explanation power of the proposed model.
Furthermore, intuitive case studies are conducted to help end users to
understand how the framework works.

positive:
Most conventional sentence similarity methods only focus on similar parts of
two input sentences, and simply ignore the dissimilar parts, which usually give
us some clues and semantic meanings about the sentences. In this work, we
propose a model to take into account both the similarities and dissimilarities
by decomposing and composing lexical semantics over sentences. The model
represents each word as a vector, and calculates a semantic matching vector for
each word based on all words in the other sentence. Then, each word vector is
decomposed into a similar component and a dissimilar component based on the
semantic matching vector. After this, a two-channel CNN model is employed to
capture features by composing the similar and dissimilar components. Finally, a
similarity score is estimated over the composed feature vectors. Experimental
results show that our model gets the state-of-the-art performance on the answer
sentence selection task, and achieves a comparable result on the paraphrase
identification task.

positive:
Systematically discovering semantic relationships in text is an important and
extensively studied area in Natural Language Processing, with various tasks
such as entailment, semantic similarity, etc. Decomposability of sentence-level
scores via subsequence alignments has been proposed as a way to make models
more interpretable. We study the problem of aligning components of sentences
leading to an interpretable model for semantic textual similarity. In this
paper, we introduce a novel pointer network based model with a sentinel gating
function to align constituent chunks, which are represented using BERT. We
improve this base model with a loss function to equally penalize misalignments
in both sentences, ensuring the alignments are bidirectional. Finally, to guide
the network with structured external knowledge, we introduce first-order logic
constraints based on ConceptNet and syntactic knowledge. The model achieves an
F1 score of 97.73 and 96.32 on the benchmark SemEval datasets for the chunk
alignment task, showing large improvements over the existing solutions. Source
code is available at
https://github.com/manishb89/interpretable_sentence_similarity

positive:
Methods for learning word representations using large text corpora have
received much attention lately due to their impressive performance in numerous
natural language processing (NLP) tasks such as, semantic similarity
measurement, and word analogy detection. Despite their success, these
data-driven word representation learning methods do not consider the rich
semantic relational structure between words in a co-occurring context. On the
other hand, already much manual effort has gone into the construction of
semantic lexicons such as the WordNet that represent the meanings of words by
defining the various relationships that exist among the words in a language. We
consider the question, can we improve the word representations learnt using a
corpora by integrating the knowledge from semantic lexicons?. For this purpose,
we propose a joint word representation learning method that simultaneously
predicts the co-occurrences of two words in a sentence subject to the
relational constrains given by the semantic lexicon. We use relations that
exist between words in the lexicon to regularize the word representations
learnt from the corpus. Our proposed method statistically significantly
outperforms previously proposed methods for incorporating semantic lexicons
into word representations on several benchmark datasets for semantic similarity
and word analogy.

positive:
The mathematical representation of semantics is a key issue for Natural
Language Processing (NLP). A lot of research has been devoted to finding ways
of representing the semantics of individual words in vector spaces.
Distributional approaches --- meaning distributed representations that exploit
co-occurrence statistics of large corpora --- have proved popular and
successful across a number of tasks. However, natural language usually comes in
structures beyond the word level, with meaning arising not only from the
individual words but also the structure they are contained in at the phrasal or
sentential level. Modelling the compositional process by which the meaning of
an utterance arises from the meaning of its parts is an equally fundamental
task of NLP.
  This dissertation explores methods for learning distributed semantic
representations and models for composing these into representations for larger
linguistic units. Our underlying hypothesis is that neural models are a
suitable vehicle for learning semantically rich representations and that such
representations in turn are suitable vehicles for solving important tasks in
natural language processing. The contribution of this thesis is a thorough
evaluation of our hypothesis, as part of which we introduce several new
approaches to representation learning and compositional semantics, as well as
multiple state-of-the-art models which apply distributed semantic
representations to various tasks in NLP.

positive:
The bag-of-words (BOW) model is the common approach for classifying
documents, where words are used as feature for training a classifier. This
generally involves a huge number of features. Some techniques, such as Latent
Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA), have been
designed to summarize documents in a lower dimension with the least semantic
information loss. Some semantic information is nevertheless always lost, since
only words are considered. Instead, we aim at using information coming from
n-grams to overcome this limitation, while remaining in a low-dimension space.
Many approaches, such as the Skip-gram model, provide good word vector
representations very quickly. We propose to average these representations to
obtain representations of n-grams. All n-grams are thus embedded in a same
semantic space. A K-means clustering can then group them into semantic
concepts. The number of features is therefore dramatically reduced and
documents can be represented as bag of semantic concepts. We show that this
model outperforms LSA and LDA on a sentiment classification task, and yields
similar results than a traditional BOW-model with far less features.

positive:
This paper proposes a neural semantic parsing approach -- Sequence-to-Action,
which models semantic parsing as an end-to-end semantic graph generation
process. Our method simultaneously leverages the advantages from two recent
promising directions of semantic parsing. Firstly, our model uses a semantic
graph to represent the meaning of a sentence, which has a tight-coupling with
knowledge bases. Secondly, by leveraging the powerful representation learning
and prediction ability of neural network models, we propose a RNN model which
can effectively map sentences to action sequences for semantic graph
generation. Experiments show that our method achieves state-of-the-art
performance on OVERNIGHT dataset and gets competitive performance on GEO and
ATIS datasets.

positive:
Attributes of words and relations between two words are central to numerous
tasks in Artificial Intelligence such as knowledge representation, similarity
measurement, and analogy detection. Often when two words share one or more
attributes in common, they are connected by some semantic relations. On the
other hand, if there are numerous semantic relations between two words, we can
expect some of the attributes of one of the words to be inherited by the other.
Motivated by this close connection between attributes and relations, given a
relational graph in which words are inter- connected via numerous semantic
relations, we propose a method to learn a latent representation for the
individual words. The proposed method considers not only the co-occurrences of
words as done by existing approaches for word representation learning, but also
the semantic relations in which two words co-occur. To evaluate the accuracy of
the word representations learnt using the proposed method, we use the learnt
word representations to solve semantic word analogy problems. Our experimental
results show that it is possible to learn better word representations by using
semantic semantics between words.

positive:
In this paper we propose a new language model called AGENT, which stands for
Adversarial Generation and Encoding of Nested Texts. AGENT is designed for
encoding, generating and refining documents that consist of a long and coherent
text, such as an entire book, provided they are hierarchically annotated
(nested). i.e. divided into sentences, paragraphs and chapters. The core idea
of our system is learning vector representations for each level of the text
hierarchy (sentences, paragraphs, etc...), and train each such representation
to perform 3 tasks: The task of reconstructing the sequence of vectors from a
lower level that was used to create the representation, and generalized
versions of the Masked Language Modeling (MLM) and "Next Sentence Prediction"
tasks from BERT Devlin et al. [2018]. Additionally we present a new adversarial
model for long text generation and suggest a way to improve the coherence of
the generated text by traversing its vector representation tree.

positive:
Unsupervised text embeddings extraction is crucial for text understanding in
machine learning. Word2Vec and its variants have received substantial success
in mapping words with similar syntactic or semantic meaning to vectors close to
each other. However, extracting context-aware word-sequence embedding remains a
challenging task. Training over large corpus is difficult as labels are
difficult to get. More importantly, it is challenging for pre-trained models to
obtain word-sequence embeddings that are universally good for all downstream
tasks or for any new datasets. We propose a two-phased ConvDic+DeconvDec
framework to solve the problem by combining a word-sequence dictionary learning
model with a word-sequence embedding decode model. We propose a convolutional
tensor decomposition mechanism to learn good word-sequence phrase dictionary in
the learning phase. It is proved to be more accurate and much more efficient
than the popular alternating minimization method. In the decode phase, we
introduce a deconvolution framework that is immune to the problem of varying
sentence lengths. The word-sequence embeddings we extracted using
ConvDic+DeconvDec are universally good for a few downstream tasks we test on.
The framework requires neither pre-training nor prior/outside information.

positive:
We design a new technique for the distributional semantic modeling with a
neural network-based approach to learn distributed term representations (or
term embeddings) - term vector space models as a result, inspired by the recent
ontology-related approach (using different types of contextual knowledge such
as syntactic knowledge, terminological knowledge, semantic knowledge, etc.) to
the identification of terms (term extraction) and relations between them
(relation extraction) called semantic pre-processing technology - SPT. Our
method relies on automatic term extraction from the natural language texts and
subsequent formation of the problem-oriented or application-oriented (also
deeply annotated) text corpora where the fundamental entity is the term
(includes non-compositional and compositional terms). This gives us an
opportunity to changeover from distributed word representations (or word
embeddings) to distributed term representations (or term embeddings). This
transition will allow to generate more accurate semantic maps of different
subject domains (also, of relations between input terms - it is useful to
explore clusters and oppositions, or to test your hypotheses about them). The
semantic map can be represented as a graph using Vec2graph - a Python library
for visualizing word embeddings (term embeddings in our case) as dynamic and
interactive graphs. The Vec2graph library coupled with term embeddings will not
only improve accuracy in solving standard NLP tasks, but also update the
conventional concept of automated ontology development. The main practical
result of our work is the development kit (set of toolkits represented as web
service APIs and web application), which provides all necessary routines for
the basic linguistic pre-processing and the semantic pre-processing of the
natural language texts in Ukrainian for future training of term vector space
models.

positive:
Neural word representations have proven useful in Natural Language Processing
(NLP) tasks due to their ability to efficiently model complex semantic and
syntactic word relationships. However, most techniques model only one
representation per word, despite the fact that a single word can have multiple
meanings or "senses". Some techniques model words by using multiple vectors
that are clustered based on context. However, recent neural approaches rarely
focus on the application to a consuming NLP algorithm. Furthermore, the
training process of recent word-sense models is expensive relative to
single-sense embedding processes. This paper presents a novel approach which
addresses these concerns by modeling multiple embeddings for each word based on
supervised disambiguation, which provides a fast and accurate way for a
consuming NLP model to select a sense-disambiguated embedding. We demonstrate
that these embeddings can disambiguate both contrastive senses such as nominal
and verbal senses as well as nuanced senses such as sarcasm. We further
evaluate Part-of-Speech disambiguated embeddings on neural dependency parsing,
yielding a greater than 8% average error reduction in unlabeled attachment
scores across 6 languages.

positive:
Many machine learning algorithms require the input to be represented as a
fixed-length feature vector. When it comes to texts, one of the most common
fixed-length features is bag-of-words. Despite their popularity, bag-of-words
features have two major weaknesses: they lose the ordering of the words and
they also ignore semantics of the words. For example, "powerful," "strong" and
"Paris" are equally distant. In this paper, we propose Paragraph Vector, an
unsupervised algorithm that learns fixed-length feature representations from
variable-length pieces of texts, such as sentences, paragraphs, and documents.
Our algorithm represents each document by a dense vector which is trained to
predict words in the document. Its construction gives our algorithm the
potential to overcome the weaknesses of bag-of-words models. Empirical results
show that Paragraph Vectors outperform bag-of-words models as well as other
techniques for text representations. Finally, we achieve new state-of-the-art
results on several text classification and sentiment analysis tasks.

positive:
Fact checking is a challenging task because verifying the truthfulness of a
claim requires reasoning about multiple retrievable evidence. In this work, we
present a method suitable for reasoning about the semantic-level structure of
evidence. Unlike most previous works, which typically represent evidence
sentences with either string concatenation or fusing the features of isolated
evidence sentences, our approach operates on rich semantic structures of
evidence obtained by semantic role labeling. We propose two mechanisms to
exploit the structure of evidence while leveraging the advances of pre-trained
models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we
first utilize the graph structure to re-define the relative distances of words,
with the intuition that semantically related words should have short distances.
Then, we adopt graph convolutional network and graph attention network to
propagate and aggregate information from neighboring nodes on the graph. We
evaluate our system on FEVER, a benchmark dataset for fact checking, and find
that rich structural information is helpful and both our graph-based mechanisms
improve the accuracy. Our model is the state-of-the-art system in terms of both
official evaluation metrics, namely claim verification accuracy and FEVER
score.

positive:
Capturing the compositional process which maps the meaning of words to that
of documents is a central challenge for researchers in Natural Language
Processing and Information Retrieval. We introduce a model that is able to
represent the meaning of documents by embedding them in a low dimensional
vector space, while preserving distinctions of word and sentence order crucial
for capturing nuanced semantics. Our model is based on an extended Dynamic
Convolution Neural Network, which learns convolution filters at both the
sentence and document level, hierarchically learning to capture and compose low
level lexical features into high level semantic concepts. We demonstrate the
effectiveness of this model on a range of document modelling tasks, achieving
strong results with no feature engineering and with a more compact model.
Inspired by recent advances in visualising deep convolution networks for
computer vision, we present a novel visualisation technique for our document
networks which not only provides insight into their learning process, but also
can be interpreted to produce a compelling automatic summarisation system for
texts.

positive:
As the first step in automated natural language processing, representing
words and sentences is of central importance and has attracted significant
research attention. Different approaches, from the early one-hot and
bag-of-words representation to more recent distributional dense and sparse
representations, were proposed. Despite the successful results that have been
achieved, such vectors tend to consist of uninterpretable components and face
nontrivial challenge in both memory and computational requirement in practical
applications. In this paper, we designed a novel representation model that
projects dense word vectors into a higher dimensional space and favors a highly
sparse and binary representation of word vectors with potentially interpretable
components, while trying to maintain pairwise inner products between original
vectors as much as possible. Computationally, our model is relaxed as a
symmetric non-negative matrix factorization problem which admits a fast yet
effective solution. In a series of empirical evaluations, the proposed model
exhibited consistent improvement and high potential in practical applications.

positive:
Vector representation of sentences is important for many text processing
tasks that involve clustering, classifying, or ranking sentences. Recently,
distributed representation of sentences learned by neural models from unlabeled
data has been shown to outperform the traditional bag-of-words representation.
However, most of these learning methods consider only the content of a sentence
and disregard the relations among sentences in a discourse by and large.
  In this paper, we propose a series of novel models for learning latent
representations of sentences (Sen2Vec) that consider the content of a sentence
as well as inter-sentence relations. We first represent the inter-sentence
relations with a language network and then use the network to induce contextual
information into the content-based Sen2Vec models. Two different approaches are
introduced to exploit the information in the network. Our first approach
retrofits (already trained) Sen2Vec vectors with respect to the network in two
different ways: (1) using the adjacency relations of a node, and (2) using a
stochastic sampling method which is more flexible in sampling neighbors of a
node. The second approach uses a regularizer to encode the information in the
network into the existing Sen2Vec model. Experimental results show that our
proposed models outperform existing methods in three fundamental information
system tasks demonstrating the effectiveness of our approach. The models
leverage the computational power of multi-core CPUs to achieve fine-grained
computational efficiency. We make our code publicly available upon acceptance.

positive:
Learning a high-dimensional dense representation for vocabulary terms, also
known as a word embedding, has recently attracted much attention in natural
language processing and information retrieval tasks. The embedding vectors are
typically learned based on term proximity in a large corpus. This means that
the objective in well-known word embedding algorithms, e.g., word2vec, is to
accurately predict adjacent word(s) for a given word or context. However, this
objective is not necessarily equivalent to the goal of many information
retrieval (IR) tasks. The primary objective in various IR tasks is to capture
relevance instead of term proximity, syntactic, or even semantic similarity.
This is the motivation for developing unsupervised relevance-based word
embedding models that learn word representations based on query-document
relevance information. In this paper, we propose two learning models with
different objective functions; one learns a relevance distribution over the
vocabulary set for each query, and the other classifies each term as belonging
to the relevant or non-relevant class for each query. To train our models, we
used over six million unique queries and the top ranked documents retrieved in
response to each query, which are assumed to be relevant to the query. We
extrinsically evaluate our learned word representation models using two IR
tasks: query expansion and query classification. Both query expansion
experiments on four TREC collections and query classification experiments on
the KDD Cup 2005 dataset suggest that the relevance-based word embedding models
significantly outperform state-of-the-art proximity-based embedding models,
such as word2vec and GloVe.

positive:
Sentence embeddings encode natural language sentences as low-dimensional
dense vectors. A great deal of effort has been put into using sentence
embeddings to improve several important natural language processing tasks.
Relation extraction is such an NLP task that aims at identifying structured
relations defined in a knowledge base from unstructured text. A promising and
more efficient approach would be to embed both the text and structured
knowledge in low-dimensional spaces and discover semantic alignments or
mappings between them. Although a number of techniques have been proposed in
the literature for embedding both sentences and knowledge graphs, little is
known about the structural and semantic properties of these embedding spaces in
terms of relation extraction. In this paper, we investigate the aforementioned
properties by evaluating the extent to which sentences carrying similar senses
are embedded in close proximity sub-spaces, and if we can exploit that
structure to align sentences to a knowledge graph. We propose a set of
experiments using a widely-used large-scale data set for relation extraction
and focusing on a set of key sentence embedding methods. We additionally
provide the code for reproducing these experiments at
https://github.com/akalino/semantic-structural-sentences. These embedding
methods cover a wide variety of techniques ranging from simple word embedding
combination to transformer-based BERT-style model. Our experimental results
show that different embedding spaces have different degrees of strength for the
structural and semantic properties. These results provide useful information
for developing embedding-based relation extraction methods.

positive:
Semantic similarity measures are an important part in Natural Language
Processing tasks. However Semantic similarity measures built for general use do
not perform well within specific domains. Therefore in this study we introduce
a domain specific semantic similarity measure that was created by the
synergistic union of word2vec, a word embedding method that is used for
semantic similarity calculation and lexicon based (lexical) semantic similarity
methods. We prove that this proposed methodology out performs word embedding
methods trained on generic corpus and methods trained on domain specific corpus
but do not use lexical semantic similarity methods to augment the results.
Further, we prove that text lemmatization can improve the performance of word
embedding methods.

positive:
Techniques in which words are represented as vectors have proved useful in
many applications in computational linguistics, however there is currently no
general semantic formalism for representing meaning in terms of vectors. We
present a framework for natural language semantics in which words, phrases and
sentences are all represented as vectors, based on a theoretical analysis which
assumes that meaning is determined by context.
  In the theoretical analysis, we define a corpus model as a mathematical
abstraction of a text corpus. The meaning of a string of words is assumed to be
a vector representing the contexts it occurs in in the corpus model. Based on
this assumption, we can show that the vector representations of words can be
considered as elements of an algebra over a field. We note that in applications
of vector spaces to representing meanings of words there is an underlying
lattice structure; we interpret the partial ordering of the lattice as
describing entailment between meanings. We also define the context-theoretic
probability of a string, and, based on this and the lattice structure, a degree
of entailment between strings.
  Together these properties form guidelines as to how to construct semantic
representations within the framework. A context theory is an implementation of
the framework; in an implementation strings are represented as vectors with the
properties deduced from the theoretical analysis.
  We show how to incorporate logical semantics into context theories; this
enables us to represent statistical information about uncertainty by taking
weighted sums of individual representations. We also use the framework to
analyse approaches to the task of recognising textual entailment, to
ontological representations of meaning and to representing syntactic structure.
For the latter, we give new algebraic descriptions of link grammar.

positive:
Distributional semantic models provide vector representations for words by
gathering co-occurrence frequencies from corpora of text. Compositional
distributional models extend these from words to phrases and sentences. In
categorical compositional distributional semantics, phrase and sentence
representations are functions of their grammatical structure and
representations of the words therein. In this setting, grammatical structures
are formalised by morphisms of a compact closed category and meanings of words
are formalised by objects of the same category. These can be instantiated in
the form of vectors or density matrices. This paper concerns the applications
of this model to phrase and sentence level entailment. We argue that
entropy-based distances of vectors and density matrices provide a good
candidate to measure word-level entailment, show the advantage of density
matrices over vectors for word level entailments, and prove that these
distances extend compositionally from words to phrases and sentences. We
exemplify our theoretical constructions on real data and a toy entailment
dataset and provide preliminary experimental evidence.

positive:
Distributional models that learn rich semantic word representations are a
success story of recent NLP research. However, developing models that learn
useful representations of phrases and sentences has proved far harder. We
propose using the definitions found in everyday dictionaries as a means of
bridging this gap between lexical and phrasal semantics. Neural language
embedding models can be effectively trained to map dictionary definitions
(phrases) to (lexical) representations of the words defined by those
definitions. We present two applications of these architectures: "reverse
dictionaries" that return the name of a concept given a definition or
description and general-knowledge crossword question answerers. On both tasks,
neural language embedding models trained on definitions from a handful of
freely-available lexical resources perform as well or better than existing
commercial systems that rely on significant task-specific engineering. The
results highlight the effectiveness of both neural embedding architectures and
definition-based training for developing models that understand phrases and
sentences.

positive:
For natural language processing systems, two kinds of evidence support the
use of text representations from neural language models "pretrained" on large
unannotated corpora: performance on application-inspired benchmarks (Peters et
al., 2018, inter alia), and the emergence of syntactic abstractions in those
representations (Tenney et al., 2019, inter alia). On the other hand, the lack
of grounded supervision calls into question how well these representations can
ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent
language models -- specifically focusing on predicate-argument structure as
operationalized by semantic dependencies (Ivanova et al., 2012) -- and find
that, unlike syntax, semantics is not brought to the surface by today's
pretrained models. We then use convolutional graph encoders to explicitly
incorporate semantic parses into task-specific finetuning, yielding benefits to
natural language understanding (NLU) tasks in the GLUE benchmark. This approach
demonstrates the potential for general-purpose (rather than task-specific)
linguistic supervision, above and beyond conventional pretraining and
finetuning. Several diagnostics help to localize the benefits of our approach.

positive:
We propose a generative model for a sentence that uses two latent variables,
with one intended to represent the syntax of the sentence and the other to
represent its semantics. We show we can achieve better disentanglement between
semantic and syntactic representations by training with multiple losses,
including losses that exploit aligned paraphrastic sentences and word-order
information. We also investigate the effect of moving from bag-of-words to
recurrent neural network modules. We evaluate our models as well as several
popular pretrained embeddings on standard semantic similarity tasks and novel
syntactic similarity tasks. Empirically, we find that the model with the best
performing syntactic and semantic representations also gives rise to the most
disentangled representations.

positive:
Predicting patient mortality is an important and challenging problem in the
healthcare domain, especially for intensive care unit (ICU) patients.
Electronic health notes serve as a rich source for learning patient
representations, that can facilitate effective risk assessment. However, a
large portion of clinical notes are unstructured and also contain domain
specific terminologies, from which we need to extract structured information.
In this paper, we introduce an embedding framework to learn
semantically-plausible distributed representations of clinical notes that
exploits the semantic correspondence between the unstructured texts and their
corresponding structured knowledge, known as semantic frame, in a hierarchical
fashion. Our approach integrates text modeling and semantic correspondence
learning into a single model that comprises 1) an unstructured embedding module
that makes use of self-similarity matrix representations in order to inject
structural regularities of different segments inherent in clinical texts to
promote local coherence, 2) a structured embedding module to embed the semantic
frames (e.g., UMLS semantic types) with deep ConvNet and 3) a hierarchical
semantic correspondence module that embeds by enhancing the interactions
between text-semantic frame embedding pairs at multiple levels (i.e., words,
sentence, note). Evaluations on multiple embedding benchmarks on post discharge
intensive care patient mortality prediction tasks demonstrate its effectiveness
compared to approaches that do not exploit the semantic interactions between
structured and unstructured information present in clinical notes.

positive:
In natural languages, words are used in association to construct sentences.
It is not words in isolation, but the appropriate combination of hierarchical
structures that conveys the meaning of the whole sentence. Neural networks can
capture expressive language features; however, insights into the link between
words and sentences are difficult to acquire automatically. In this work, we
design a deep neural network architecture that explicitly wires lower and
higher linguistic components; we then evaluate its ability to perform the same
task at different hierarchical levels. Settling on broad text classification
tasks, we show that our model, MHAL, learns to simultaneously solve them at
different levels of granularity by fluidly transferring knowledge between
hierarchies. Using a multi-head attention mechanism to tie the representations
between single words and full sentences, MHAL systematically outperforms
equivalent models that are not incentivized towards developing compositional
representations. Moreover, we demonstrate that, with the proposed architecture,
the sentence information flows naturally to individual words, allowing the
model to behave like a sequence labeller (which is a lower, word-level task)
even without any word supervision, in a zero-shot fashion.

positive:
Semantic sentence embedding models encode natural language sentences into
vectors, such that closeness in embedding space indicates closeness in the
semantics between the sentences. Bilingual data offers a useful signal for
learning such embeddings: properties shared by both sentences in a translation
pair are likely semantic, while divergent properties are likely stylistic or
language-specific. We propose a deep latent variable model that attempts to
perform source separation on parallel sentences, isolating what they have in
common in a latent semantic vector, and explaining what is left over with
language-specific latent vectors. Our proposed approach differs from past work
on semantic sentence encoding in two ways. First, by using a variational
probabilistic framework, we introduce priors that encourage source separation,
and can use our model's posterior to predict sentence embeddings for
monolingual data at test time. Second, we use high-capacity transformers as
both data generating distributions and inference networks -- contrasting with
most past work on sentence embeddings. In experiments, our approach
substantially outperforms the state-of-the-art on a standard suite of
unsupervised semantic similarity evaluations. Further, we demonstrate that our
approach yields the largest gains on more difficult subsets of these
evaluations where simple word overlap is not a good indicator of similarity.

positive:
An important component of achieving language understanding is mastering the
composition of sentence meaning, but an immediate challenge to solving this
problem is the opacity of sentence vector representations produced by current
neural sentence composition models. We present a method to address this
challenge, developing tasks that directly target compositional meaning
information in sentence vector representations with a high degree of precision
and control. To enable the creation of these controlled tasks, we introduce a
specialized sentence generation system that produces large, annotated sentence
sets meeting specified syntactic, semantic and lexical constraints. We describe
the details of the method and generation system, and then present results of
experiments applying our method to probe for compositional information in
embeddings from a number of existing sentence composition models. We find that
the method is able to extract useful information about the differing capacities
of these models, and we discuss the implications of our results with respect to
these systems' capturing of sentence information. We make available for public
use the datasets used for these experiments, as well as the generation system.

negative:
Natural language generation of coherent long texts like paragraphs or longer
documents is a challenging problem for recurrent networks models. In this
paper, we explore an important step toward this generation task: training an
LSTM (Long-short term memory) auto-encoder to preserve and reconstruct
multi-sentence paragraphs. We introduce an LSTM model that hierarchically
builds an embedding for a paragraph from embeddings for sentences and words,
then decodes this embedding to reconstruct the original paragraph. We evaluate
the reconstructed paragraph using standard metrics like ROUGE and Entity Grid,
showing that neural models are able to encode texts in a way that preserve
syntactic, semantic, and discourse coherence. While only a first step toward
generating coherent text units from neural models, our work has the potential
to significantly impact natural language generation and
summarization\footnote{Code for the three models described in this paper can be
found at www.stanford.edu/~jiweil/ .

negative:
Dense word embeddings, which encode semantic meanings of words to low
dimensional vector spaces have become very popular in natural language
processing (NLP) research due to their state-of-the-art performances in many
NLP tasks. Word embeddings are substantially successful in capturing semantic
relations among words, so a meaningful semantic structure must be present in
the respective vector spaces. However, in many cases, this semantic structure
is broadly and heterogeneously distributed across the embedding dimensions,
which makes interpretation a big challenge. In this study, we propose a
statistical method to uncover the latent semantic structure in the dense word
embeddings. To perform our analysis we introduce a new dataset (SEMCAT) that
contains more than 6500 words semantically grouped under 110 categories. We
further propose a method to quantify the interpretability of the word
embeddings; the proposed method is a practical alternative to the classical
word intrusion test that requires human intervention.

negative:
Word2vec (Mikolov et al., 2013) has proven to be successful in natural
language processing by capturing the semantic relationships between different
words. Built on top of single-word embeddings, paragraph vectors (Le and
Mikolov, 2014) find fixed-length representations for pieces of text with
arbitrary lengths, such as documents, paragraphs, and sentences. In this work,
we propose a novel interpretation for neural-network-based paragraph vectors by
developing an unsupervised generative model whose maximum likelihood solution
corresponds to traditional paragraph vectors. This probabilistic formulation
allows us to go beyond point estimates of parameters and to perform Bayesian
posterior inference. We find that the entropy of paragraph vectors decreases
with the length of documents, and that information about posterior uncertainty
improves performance in supervised learning tasks such as sentiment analysis
and paraphrase detection.

negative:
Recent progress on image captioning has made it possible to generate novel
sentences describing images in natural language, but compressing an image into
a single sentence can describe visual content in only coarse detail. While one
new captioning approach, dense captioning, can potentially describe images in
finer levels of detail by captioning many regions within an image, it in turn
is unable to produce a coherent story for an image. In this paper we overcome
these limitations by generating entire paragraphs for describing images, which
can tell detailed, unified stories. We develop a model that decomposes both
images and paragraphs into their constituent parts, detecting semantic regions
in images and using a hierarchical recurrent neural network to reason about
language. Linguistic analysis confirms the complexity of the paragraph
generation task, and thorough experiments on a new dataset of image and
paragraph pairs demonstrate the effectiveness of our approach.

negative:
Sentence embedding is an important research topic in natural language
processing. It is essential to generate a good embedding vector that fully
reflects the semantic meaning of a sentence in order to achieve an enhanced
performance for various natural language processing tasks, such as machine
translation and document classification. Thus far, various sentence embedding
models have been proposed, and their feasibility has been demonstrated through
good performances on tasks following embedding, such as sentiment analysis and
sentence classification. However, because the performances of sentence
classification and sentiment analysis can be enhanced by using a simple
sentence representation method, it is not sufficient to claim that these models
fully reflect the meanings of sentences based on good performances for such
tasks. In this paper, inspired by human language recognition, we propose the
following concept of semantic coherence, which should be satisfied for a good
sentence embedding method: similar sentences should be located close to each
other in the embedding space. Then, we propose the Paraphrase-Thought
(P-thought) model to pursue semantic coherence as much as possible.
Experimental results on two paraphrase identification datasets (MS COCO and STS
benchmark) show that the P-thought models outperform the benchmarked sentence
embedding methods.

negative:
Word vector representations are central to deep learning natural language
processing models. Many forms of these vectors, known as embeddings, exist,
including word2vec and GloVe. Embeddings are trained on large corpora and learn
the word's usage in context, capturing the semantic relationship between words.
However, the semantics from such training are at the level of distinct words
(known as word types), and can be ambiguous when, for example, a word type can
be either a noun or a verb. In question answering, parts-of-speech and named
entity types are important, but encoding these attributes in neural models
expands the size of the input. This research employs holographic compression of
pre-trained embeddings, to represent a token, its part-of-speech, and named
entity type, in the same dimension as representing only the token. The
implementation, in a modified question answering recurrent deep learning
network, shows that semantic relationships are preserved, and yields strong
performance.

negative:
Sentence representation at the semantic level is a challenging task for
Natural Language Processing and Artificial Intelligence. Despite the advances
in word embeddings (i.e. word vector representations), capturing sentence
meaning is an open question due to complexities of semantic interactions among
words. In this paper, we present an embedding method, which is aimed at
learning unsupervised sentence representations from unlabeled text. We propose
an unsupervised method that models a sentence as a weighted series of word
embeddings. The weights of the word embeddings are fitted by using Shannon's
word entropies provided by the Term Frequency--Inverse Document Frequency
(TF--IDF) transform. The hyperparameters of the model can be selected according
to the properties of data (e.g. sentence length and textual gender).
Hyperparameter selection involves word embedding methods and dimensionalities,
as well as weighting schemata. Our method offers advantages over existing
methods: identifiable modules, short-term training, online inference of
(unseen) sentence representations, as well as independence from domain,
external knowledge and language resources. Results showed that our model
outperformed the state of the art in well-known Semantic Textual Similarity
(STS) benchmarks. Moreover, our model reached state-of-the-art performance when
compared to supervised and knowledge-based STS systems.

negative:
Word embeddings are rich word representations, which in combination with deep
neural networks, lead to large performance gains for many NLP tasks. However,
word embeddings are represented by dense, real-valued vectors and they are
therefore not directly interpretable. Thus, computational operations based on
them are also not well understood. In this paper, we present an approach for
analyzing structures in the semantic vector space to get a better understanding
of the underlying semantic encoding principles. We present a framework for
decomposing word embeddings into smaller meaningful units which we call
sub-vectors. The framework opens up a wide range of possibilities analyzing
phenomena in vector space semantics, as well as solving concrete NLP problems:
We introduce the category completion task and show that a sub-vector based
approach is superior to supervised techniques; We present a sub-vector based
method for solving the word analogy task, which substantially outperforms
different variants of the traditional vector-offset method.

negative:
This paper introduces a sentence to vector encoding framework suitable for
advanced natural language processing. Our latent representation is shown to
encode sentences with common semantic information with similar vector
representations. The vector representation is extracted from an encoder-decoder
model which is trained on sentence paraphrase pairs. We demonstrate the
application of the sentence representations for two different tasks -- sentence
paraphrasing and paragraph summarization, making it attractive for commonly
used recurrent frameworks that process text. Experimental results help gain
insight how vector representations are suitable for advanced language
embedding.

negative:
In this paper, we present Hierarchical Graph Network (HGN) for multi-hop
question answering. To aggregate clues from scattered texts across multiple
paragraphs, a hierarchical graph is created by constructing nodes on different
levels of granularity (questions, paragraphs, sentences, entities), the
representations of which are initialized with pre-trained contextual encoders.
Given this hierarchical graph, the initial node representations are updated
through graph propagation, and multi-hop reasoning is performed via traversing
through the graph edges for each subsequent sub-task (e.g., paragraph
selection, supporting facts extraction, answer prediction). By weaving
heterogeneous nodes into an integral unified graph, this hierarchical
differentiation of node granularity enables HGN to support different question
answering sub-tasks simultaneously. Experiments on the HotpotQA benchmark
demonstrate that the proposed model achieves new state of the art,
outperforming existing multi-hop QA approaches.

negative:
Recently, there has been a lot of effort to represent words in continuous
vector spaces. Those representations have been shown to capture both semantic
and syntactic information about words. However, distributed representations of
phrases remain a challenge. We introduce a novel model that jointly learns word
vector representations and their summation. Word representations are learnt
using the word co-occurrence statistical information. To embed sequences of
words (i.e. phrases) with different sizes into a common semantic space, we
propose to average word vector representations. In contrast with previous
methods which reported a posteriori some compositionality aspects by simple
summation, we simultaneously train words to sum, while keeping the maximum
information from the original vectors. We evaluate the quality of the word
representations on several classical word evaluation tasks, and we introduce a
novel task to evaluate the quality of the phrase representations. While our
distributed representations compete with other methods of learning word
representations on word evaluations, we show that they give better performance
on the phrase evaluation. Such representations of phrases could be interesting
for many tasks in natural language processing.

negative:
Grounding referring expressions in images aims to locate the object instance
in an image described by a referring expression. It involves a joint
understanding of natural language and image content, and is essential for a
range of visual tasks related to human-computer interaction. As a
language-to-vision matching task, the core of this problem is to not only
extract all the necessary information (i.e., objects and the relationships
among them) in both the image and referring expression, but also make full use
of context information to align cross-modal semantic concepts in the extracted
information. Unfortunately, existing work on grounding referring expressions
fails to accurately extract multi-order relationships from the referring
expression and associate them with the objects and their related contexts in
the image. In this paper, we propose a Cross-Modal Relationship Extractor
(CMRE) to adaptively highlight objects and relationships (spatial and semantic
relations) related to the given expression with a cross-modal attention
mechanism, and represent the extracted information as a language-guided visual
relation graph. In addition, we propose a Gated Graph Convolutional Network
(GGCN) to compute multimodal semantic contexts by fusing information from
different modes and propagating multimodal information in the structured
relation graph. Experimental results on three common benchmark datasets show
that our Cross-Modal Relationship Inference Network, which consists of CMRE and
GGCN, significantly surpasses all existing state-of-the-art methods. Code is
available at https://github.com/sibeiyang/sgmn/tree/master/lib/cmrin_models

negative:
Recently proposed Skip-gram model is a powerful method for learning
high-dimensional word representations that capture rich semantic relationships
between words. However, Skip-gram as well as most prior work on learning word
representations does not take into account word ambiguity and maintain only
single representation per word. Although a number of Skip-gram modifications
were proposed to overcome this limitation and learn multi-prototype word
representations, they either require a known number of word meanings or learn
them using greedy heuristic approaches. In this paper we propose the Adaptive
Skip-gram model which is a nonparametric Bayesian extension of Skip-gram
capable to automatically learn the required number of representations for all
words at desired semantic resolution. We derive efficient online variational
learning algorithm for the model and empirically demonstrate its efficiency on
word-sense induction task.

negative:
Over the last few years, machine learning over graph structures has
manifested a significant enhancement in text mining applications such as event
detection, opinion mining, and news recommendation. One of the primary
challenges in this regard is structuring a graph that encodes and encompasses
the features of textual data for the effective machine learning algorithm.
Besides, exploration and exploiting of semantic relations is regarded as a
principal step in text mining applications. However, most of the traditional
text mining methods perform somewhat poor in terms of employing such relations.
In this paper, we propose a sentence-level graph-based text representation
which includes stop words to consider semantic and term relations. Then, we
employ a representation learning approach on the combined graphs of sentences
to extract the latent and continuous features of the documents. Eventually, the
learned features of the documents are fed into a deep neural network for the
sentiment classification task. The experimental results demonstrate that the
proposed method substantially outperforms the related sentiment analysis
approaches based on several benchmark datasets. Furthermore, our method can be
generalized on different datasets without any dependency on pre-trained word
embeddings.

negative:
Natural language processing often involves computations with semantic or
syntactic graphs to facilitate sophisticated reasoning based on structural
relationships. While convolution kernels provide a powerful tool for comparing
graph structure based on node (word) level relationships, they are difficult to
customize and can be computationally expensive. We propose a generalization of
convolution kernels, with a nonstationary model, for better expressibility of
natural languages in supervised settings. For a scalable learning of the
parameters introduced with our model, we propose a novel algorithm that
leverages stochastic sampling on k-nearest neighbor graphs, along with
approximations based on locality-sensitive hashing. We demonstrate the
advantages of our approach on a challenging real-world (structured inference)
problem of automatically extracting biological models from the text of
scientific papers.

negative:
As an efficient model for knowledge organization, the knowledge graph has
been widely adopted in several fields, e.g., biomedicine, sociology, and
education. And there is a steady trend of learning embedding representations of
knowledge graphs to facilitate knowledge graph construction and downstream
tasks. In general, knowledge graph embedding techniques aim to learn vectorized
representations which preserve the structural information of the graph. And
conventional embedding learning models rely on structural relationships among
entities and relations. However, in educational knowledge graphs, structural
relationships are not the focus. Instead, rich literals of the graphs are more
valuable. In this paper, we focus on this problem and propose a novel model for
embedding learning of educational knowledge graphs. Our model considers both
structural and literal information and jointly learns embedding
representations. Three experimental graphs were constructed based on an
educational knowledge graph which has been applied in real-world teaching. We
conducted two experiments on the three graphs and other common benchmark
graphs. The experimental results proved the effectiveness of our model and its
superiority over other baselines when processing educational knowledge graphs.

negative:
Natural language processing models have attracted much interest in the deep
learning community. This branch of study is composed of some applications such
as machine translation, sentiment analysis, named entity recognition, question
and answer, and others. Word embeddings are continuous word representations,
they are an essential module for those applications and are generally used as
input word representation to the deep learning models. Word2Vec and GloVe are
two popular methods to learn word embeddings. They achieve good word
representations, however, they learn representations with limited information
because they ignore the morphological information of the words and consider
only one representation vector for each word. This approach implies that
Word2Vec and GloVe are unaware of the word inner structure. To mitigate this
problem, the FastText model represents each word as a bag of characters
n-grams. Hence, each n-gram has a continuous vector representation, and the
final word representation is the sum of its characters n-grams vectors.
Nevertheless, the use of all n-grams character of a word is a poor approach
since some n-grams have no semantic relation with their words and increase the
amount of potentially useless information. This approach also increases the
training phase time. In this work, we propose a new method for training word
embeddings, and its goal is to replace the FastText bag of character n-grams
for a bag of word morphemes through the morphological analysis of the word.
Thus, words with similar context and morphemes are represented by vectors close
to each other. To evaluate our new approach, we performed intrinsic evaluations
considering 15 different tasks, and the results show a competitive performance
compared to FastText.

negative:
Distributed representations of meaning are a natural way to encode covariance
relationships between words and phrases in NLP. By overcoming data sparsity
problems, as well as providing information about semantic relatedness which is
not available in discrete representations, distributed representations have
proven useful in many NLP tasks. Recent work has shown how compositional
semantic representations can successfully be applied to a number of monolingual
applications such as sentiment analysis. At the same time, there has been some
initial success in work on learning shared word-level representations across
languages. We combine these two approaches by proposing a method for learning
distributed representations in a multilingual setup. Our model learns to assign
similar embeddings to aligned sentences and dissimilar ones to sentence which
are not aligned while not requiring word alignments. We show that our
representations are semantically informative and apply them to a cross-lingual
document classification task where we outperform the previous state of the art.
Further, by employing parallel corpora of multiple language pairs we find that
our model learns representations that capture semantic relationships across
languages for which no parallel data was used.

negative:
Due to the increasing amount of data on the internet, finding a
highly-informative, low-dimensional representation for text is one of the main
challenges for efficient natural language processing tasks including text
classification. This representation should capture the semantic information of
the text while retaining their relevance level for document classification.
This approach maps the documents with similar topics to a similar space in
vector space representation. To obtain representation for large text, we
propose the utilization of deep Siamese neural networks. To embed document
relevance in topics in the distributed representation, we use a Siamese neural
network to jointly learn document representations. Our Siamese network consists
of two sub-network of multi-layer perceptron. We examine our representation for
the text categorization task on BBC news dataset. The results show that the
proposed representations outperform the conventional and state-of-the-art
representations in the text classification task on this dataset.

negative:
The field of natural language understanding has experienced exponential
progress in the last few years, with impressive results in several tasks. This
success has motivated researchers to study the underlying knowledge encoded by
these models. Despite this, attempts to understand their semantic capabilities
have not been successful, often leading to non-conclusive, or contradictory
conclusions among different works. Via a probing classifier, we extract the
underlying knowledge graph of nine of the most influential language models of
the last years, including word embeddings, text generators, and context
encoders. This probe is based on concept relatedness, grounded on WordNet. Our
results reveal that all the models encode this knowledge, but suffer from
several inaccuracies. Furthermore, we show that the different architectures and
training strategies lead to different model biases. We conduct a systematic
evaluation to discover specific factors that explain why some concepts are
challenging. We hope our insights will motivate the development of models that
capture concepts more precisely.

negative:
Current approaches to learning vector representations of text that are
compatible between different languages usually require some amount of parallel
text, aligned at word, sentence or at least document level. We hypothesize
however, that different natural languages share enough semantic structure that
it should be possible, in principle, to learn compatible vector representations
just by analyzing the monolingual distribution of words.
  In order to evaluate this hypothesis, we propose a scheme to map word vectors
trained on a source language to vectors semantically compatible with word
vectors trained on a target language using an adversarial autoencoder.
  We present preliminary qualitative results and discuss possible future
developments of this technique, such as applications to cross-lingual sentence
representations.

negative:
The recently introduced continuous Skip-gram model is an efficient method for
learning high-quality distributed vector representations that capture a large
number of precise syntactic and semantic word relationships. In this paper we
present several extensions that improve both the quality of the vectors and the
training speed. By subsampling of the frequent words we obtain significant
speedup and also learn more regular word representations. We also describe a
simple alternative to the hierarchical softmax called negative sampling. An
inherent limitation of word representations is their indifference to word order
and their inability to represent idiomatic phrases. For example, the meanings
of "Canada" and "Air" cannot be easily combined to obtain "Air Canada".
Motivated by this example, we present a simple method for finding phrases in
text, and show that learning good vector representations for millions of
phrases is possible.

negative:
Semantic neologisms (SN) are defined as words that acquire a new word meaning
while maintaining their form. Given the nature of this kind of neologisms, the
task of identifying these new word meanings is currently performed manually by
specialists at observatories of neology. To detect SN in a semi-automatic way,
we developed a system that implements a combination of the following
strategies: topic modeling, keyword extraction, and word sense disambiguation.
The role of topic modeling is to detect the themes that are treated in the
input text. Themes within a text give clues about the particular meaning of the
words that are used, for example: viral has one meaning in the context of
computer science (CS) and another when talking about health. To extract
keywords, we used TextRank with POS tag filtering. With this method, we can
obtain relevant words that are already part of the Spanish lexicon. We use a
deep learning model to determine if a given keyword could have a new meaning.
Embeddings that are different from all the known meanings (or topics) indicate
that a word might be a valid SN candidate. In this study, we examine the
following word embedding models: Word2Vec, Sense2Vec, and FastText. The models
were trained with equivalent parameters using Wikipedia in Spanish as corpora.
Then we used a list of words and their concordances (obtained from our database
of neologisms) to show the different embeddings that each model yields.
Finally, we present a comparison of these outcomes with the concordances of
each word to show how we can determine if a word could be a valid candidate for
SN.

negative:
The words of a language reflect the structure of the human mind, allowing us
to transmit thoughts between individuals. However, language can represent only
a subset of our rich and detailed cognitive architecture. Here, we ask what
kinds of common knowledge (semantic memory) are captured by word meanings
(lexical semantics). We examine a prominent computational model that represents
words as vectors in a multidimensional space, such that proximity between
word-vectors approximates semantic relatedness. Because related words appear in
similar contexts, such spaces - called "word embeddings" - can be learned from
patterns of lexical co-occurrences in natural language. Despite their
popularity, a fundamental concern about word embeddings is that they appear to
be semantically "rigid": inter-word proximity captures only overall similarity,
yet human judgments about object similarities are highly context-dependent and
involve multiple, distinct semantic features. For example, dolphins and
alligators appear similar in size, but differ in intelligence and
aggressiveness. Could such context-dependent relationships be recovered from
word embeddings? To address this issue, we introduce a powerful, domain-general
solution: "semantic projection" of word-vectors onto lines that represent
various object features, like size (the line extending from the word "small" to
"big"), intelligence (from "dumb" to "smart"), or danger (from "safe" to
"dangerous"). This method, which is intuitively analogous to placing objects
"on a mental scale" between two extremes, recovers human judgments across a
range of object categories and properties. We thus show that word embeddings
inherit a wealth of common knowledge from word co-occurrence statistics and can
be flexibly manipulated to express context-dependent meanings.

negative:
Methods for learning word sense embeddings represent a single word with
multiple sense-specific vectors. These methods should not only produce
interpretable sense embeddings, but should also learn how to select which sense
to use in a given context. We propose an unsupervised model that learns sense
embeddings using a modified Gumbel softmax function, which allows for
differentiable discrete sense selection. Our model produces sense embeddings
that are competitive (and sometimes state of the art) on multiple similarity
based downstream evaluations. However, performance on these downstream
evaluations tasks does not correlate with interpretability of sense embeddings,
as we discover through an interpretability comparison with competing
multi-sense embeddings. While many previous approaches perform well on
downstream evaluations, they do not produce interpretable embeddings and learn
duplicated sense groups; our method achieves the best of both worlds.

negative:
Word-vector representations associate a high dimensional real-vector to every
word from a corpus. Recently, neural-network based methods have been proposed
for learning this representation from large corpora. This type of
word-to-vector embedding is able to keep, in the learned vector space, some of
the syntactic and semantic relationships present in the original word corpus.
This, in turn, serves to address different types of language classification
tasks by doing algebraic operations defined on the vectors. The general
practice is to assume that the semantic relationships between the words can be
inferred by the application of a-priori specified algebraic operations. Our
general goal in this paper is to show that it is possible to learn methods for
word composition in semantic spaces. Instead of expressing the compositional
method as an algebraic operation, we will encode it as a program, which can be
linear, nonlinear, or involve more intricate expressions. More remarkably, this
program will be evolved from a set of initial random programs by means of
genetic programming (GP). We show that our method is able to reproduce the same
behavior as human-designed algebraic operators. Using a word analogy task as
benchmark, we also show that GP-generated programs are able to obtain accuracy
values above those produced by the commonly used human-designed rule for
algebraic manipulation of word vectors. Finally, we show the robustness of our
approach by executing the evolved programs on the word2vec GoogleNews vectors,
learned over 3 billion running words, and assessing their accuracy in the same
word analogy task.

negative:
Retrieving information from correlative paragraphs or documents to answer
open-domain multi-hop questions is very challenging. To deal with this
challenge, most of the existing works consider paragraphs as nodes in a graph
and propose graph-based methods to retrieve them. However, in this paper, we
point out the intrinsic defect of such methods. Instead, we propose a new
architecture that models paragraphs as sequential data and considers multi-hop
information retrieval as a kind of sequence labeling task. Specifically, we
design a rewritable external memory to model the dependency among paragraphs.
Moreover, a threshold gate mechanism is proposed to eliminate the distraction
of noise paragraphs. We evaluate our method on both full wiki and distractor
subtask of HotpotQA, a public textual multi-hop QA dataset requiring multi-hop
information retrieval. Experiments show that our method achieves significant
improvement over the published state-of-the-art method in retrieval and
downstream QA task performance.

negative:
This paper presents a new model for word sense disambiguation formulated in
terms of evolutionary game theory, where each word to be disambiguated is
represented as a node on a graph whose edges represent word relations and
senses are represented as classes. The words simultaneously update their class
membership preferences according to the senses that neighboring words are
likely to choose. We use distributional information to weigh the influence that
each word has on the decisions of the others and semantic similarity
information to measure the strength of compatibility among the choices. With
this information we can formulate the word sense disambiguation problem as a
constraint satisfaction problem and solve it using tools derived from game
theory, maintaining the textual coherence. The model is based on two ideas:
similar words should be assigned to similar classes and the meaning of a word
does not depend on all the words in a text but just on some of them. The paper
provides an in-depth motivation of the idea of modeling the word sense
disambiguation problem in terms of game theory, which is illustrated by an
example. The conclusion presents an extensive analysis on the combination of
similarity measures to use in the framework and a comparison with
state-of-the-art systems. The results show that our model outperforms
state-of-the-art algorithms and can be applied to different tasks and in
different scenarios.

negative:
In the context of natural language processing, representation learning has
emerged as a newly active research subject because of its excellent performance
in many applications. Learning representations of words is a pioneering study
in this school of research. However, paragraph (or sentence and document)
embedding learning is more suitable/reasonable for some tasks, such as
sentiment classification and document summarization. Nevertheless, as far as we
are aware, there is relatively less work focusing on the development of
unsupervised paragraph embedding methods. Classic paragraph embedding methods
infer the representation of a given paragraph by considering all of the words
occurring in the paragraph. Consequently, those stop or function words that
occur frequently may mislead the embedding learning process to produce a misty
paragraph representation. Motivated by these observations, our major
contributions in this paper are twofold. First, we propose a novel unsupervised
paragraph embedding method, named the essence vector (EV) model, which aims at
not only distilling the most representative information from a paragraph but
also excluding the general background information to produce a more informative
low-dimensional vector representation for the paragraph. Second, in view of the
increasing importance of spoken content processing, an extension of the EV
model, named the denoising essence vector (D-EV) model, is proposed. The D-EV
model not only inherits the advantages of the EV model but also can infer a
more robust representation for a given spoken paragraph against imperfect
speech recognition.

negative:
Neural word representations are at the core of many state-of-the-art natural
language processing models. A widely used approach is to pre-train, store and
look up word or character embedding matrices. While useful, such
representations occupy huge memory making it hard to deploy on-device and often
do not generalize to unknown words due to vocabulary pruning.
  In this paper, we propose a skip-gram based architecture coupled with
Locality-Sensitive Hashing (LSH) projections to learn efficient dynamically
computable representations. Our model does not need to store lookup tables as
representations are computed on-the-fly and require low memory footprint. The
representations can be trained in an unsupervised fashion and can be easily
transferred to other NLP tasks. For qualitative evaluation, we analyze the
nearest neighbors of the word representations and discover semantically similar
words even with misspellings. For quantitative evaluation, we plug our
transferable projections into a simple LSTM and run it on multiple NLP tasks
and show how our transferable projections achieve better performance compared
to prior work.

negative:
Understanding the semantic relationships between terms is a fundamental task
in natural language processing applications. While structured resources that
can express those relationships in a formal way, such as ontologies, are still
scarce, a large number of linguistic resources gathering dictionary definitions
is becoming available, but understanding the semantic structure of natural
language definitions is fundamental to make them useful in semantic
interpretation tasks. Based on an analysis of a subset of WordNet's glosses, we
propose a set of semantic roles that compose the semantic structure of a
dictionary definition, and show how they are related to the definition's
syntactic configuration, identifying patterns that can be used in the
development of information extraction frameworks and semantic models.

negative:
Common language models typically predict the next word given the context. In
this work, we propose a method that improves language modeling by learning to
align the given context and the following phrase. The model does not require
any linguistic annotation of phrase segmentation. Instead, we define syntactic
heights and phrase segmentation rules, enabling the model to automatically
induce phrases, recognize their task-specific heads, and generate phrase
embeddings in an unsupervised learning manner. Our method can easily be applied
to language models with different network architectures since an independent
module is used for phrase induction and context-phrase alignment, and no change
is required in the underlying language modeling network. Experiments have shown
that our model outperformed several strong baseline models on different data
sets. We achieved a new state-of-the-art performance of 17.4 perplexity on the
Wikitext-103 dataset. Additionally, visualizing the outputs of the phrase
induction module showed that our model is able to learn approximate
phrase-level structural knowledge without any annotation.

negative:
Natural language processing (NLP) tasks tend to suffer from a paucity of
suitably annotated training data, hence the recent success of transfer learning
across a wide variety of them. The typical recipe involves: (i) training a
deep, possibly bidirectional, neural network with an objective related to
language modeling, for which training data is plentiful; and (ii) using the
trained network to derive contextual representations that are far richer than
standard linear word embeddings such as word2vec, and thus result in important
gains. In this work, we wonder whether the opposite perspective is also true:
can contextual representations trained for different NLP tasks improve language
modeling itself? Since language models (LMs) are predominantly locally
optimized, other NLP tasks may help them make better predictions based on the
entire semantic fabric of a document. We test the performance of several types
of pre-trained embeddings in neural LMs, and we investigate whether it is
possible to make the LM more aware of global semantic information through
embeddings pre-trained with a domain classification model. Initial experiments
suggest that as long as the proper objective criterion is used during training,
pre-trained embeddings are likely to be beneficial for neural language
modeling.

negative:
A neural language model trained on a text corpus can be used to induce
distributed representations of words, such that similar words end up with
similar representations. If the corpus is multilingual, the same model can be
used to learn distributed representations of languages, such that similar
languages end up with similar representations. We show that this holds even
when the multilingual corpus has been translated into English, by picking up
the faint signal left by the source languages. However, just like it is a
thorny problem to separate semantic from syntactic similarity in word
representations, it is not obvious what type of similarity is captured by
language representations. We investigate correlations and causal relationships
between language representations learned from translations on one hand, and
genetic, geographical, and several levels of structural similarity between
languages on the other. Of these, structural similarity is found to correlate
most strongly with language representation similarity, while genetic
relationships---a convenient benchmark used for evaluation in previous
work---appears to be a confounding factor. Apart from implications about
translation effects, we see this more generally as a case where NLP and
linguistic typology can interact and benefit one another.

negative:
Textbook Question Answering (TQA) is a task that one should answer a
diagram/non-diagram question given a large multi-modal context consisting of
abundant essays and diagrams. We argue that the explainability of this task
should place students as a key aspect to be considered. To address this issue,
we devise a novel architecture towards span-level eXplanations of the TQA
(XTQA) based on our proposed coarse-to-fine grained algorithm, which can
provide not only the answers but also the span-level evidences to choose them
for students. This algorithm first coarsely chooses top $M$ paragraphs relevant
to questions using the TF-IDF method, and then chooses top $K$ evidence spans
finely from all candidate spans within these paragraphs by computing the
information gain of each span to questions. Experimental results shows that
XTQA significantly improves the state-of-the-art performance compared with
baselines. The source code is available at
https://github.com/keep-smile-001/opentqa

negative:
The growing complexity of legal cases has lead to an increasing interest in
legal information retrieval systems that can effectively satisfy user-specific
information needs. However, such downstream systems typically require documents
to be properly formatted and segmented, which is often done with relatively
simple pre-processing steps, disregarding topical coherence of segments.
Systems generally rely on representations of individual sentences or
paragraphs, which may lack crucial context, or document-level representations,
which are too long for meaningful search results. To address this issue, we
propose a segmentation system that can predict topical coherence of sequential
text segments spanning several paragraphs, effectively segmenting a document
and providing a more balanced representation for downstream applications. We
build our model on top of popular transformer networks and formulate structural
text segmentation as topical change detection, by performing a series of
independent classifications that allow for efficient fine-tuning on
task-specific data. We crawl a novel dataset consisting of roughly $74,000$
online Terms-of-Service documents, including hierarchical topic annotations,
which we use for training. Results show that our proposed system significantly
outperforms baselines, and adapts well to structural peculiarities of legal
documents. We release both data and trained models to the research community
for future work.https://github.com/dennlinger/TopicalChange

negative:
This paper is concerned with the task of multi-hop open-domain Question
Answering (QA). This task is particularly challenging since it requires the
simultaneous performance of textual reasoning and efficient searching. We
present a method for retrieving multiple supporting paragraphs, nested amidst a
large knowledge base, which contain the necessary evidence to answer a given
question. Our method iteratively retrieves supporting paragraphs by forming a
joint vector representation of both a question and a paragraph. The retrieval
is performed by considering contextualized sentence-level representations of
the paragraphs in the knowledge source. Our method achieves state-of-the-art
performance over two well-known datasets, SQuAD-Open and HotpotQA, which serve
as our single- and multi-hop open-domain QA benchmarks, respectively.

negative:
Analysis of word embedding properties to inform their use in downstream NLP
tasks has largely been studied by assessing nearest neighbors. However,
geometric properties of the continuous feature space contribute directly to the
use of embedding features in downstream models, and are largely unexplored. We
consider four properties of word embedding geometry, namely: position relative
to the origin, distribution of features in the vector space, global pairwise
distances, and local pairwise distances. We define a sequence of
transformations to generate new embeddings that expose subsets of these
properties to downstream models and evaluate change in task performance to
understand the contribution of each property to NLP models. We transform
publicly available pretrained embeddings from three popular toolkits (word2vec,
GloVe, and FastText) and evaluate on a variety of intrinsic tasks, which model
linguistic information in the vector space, and extrinsic tasks, which use
vectors as input to machine learning models. We find that intrinsic evaluations
are highly sensitive to absolute position, while extrinsic tasks rely primarily
on local similarity. Our findings suggest that future embedding models and
post-processing techniques should focus primarily on similarity to nearby
points in vector space.

negative:
Semantic vector embedding techniques have proven useful in learning semantic
representations of data across multiple domains. A key application enabled by
such techniques is the ability to measure semantic similarity between given
data samples and find data most similar to a given sample. State-of-the-art
embedding approaches assume all data is available on a single site. However, in
many business settings, data is distributed across multiple edge locations and
cannot be aggregated due to a variety of constraints. Hence, the applicability
of state-of-the-art embedding approaches is limited to freely shared datasets,
leaving out applications with sensitive or mission-critical data. This paper
addresses this gap by proposing novel unsupervised algorithms called
\emph{SEEC} for learning and applying semantic vector embedding in a variety of
distributed settings. Specifically, for scenarios where multiple edge locations
can engage in joint learning, we adapt the recently proposed federated learning
techniques for semantic vector embedding. Where joint learning is not possible,
we propose novel semantic vector translation algorithms to enable semantic
query across multiple edge locations, each with its own semantic vector-space.
Experimental results on natural language as well as graph datasets show that
this may be a promising new direction.

negative:
Though there are some works on improving distributed word representations
using lexicons, the improper overfitting of the words that have multiple
meanings is a remaining issue deteriorating the learning when lexicons are
used, which needs to be solved. An alternative method is to allocate a vector
per sense instead of a vector per word. However, the word representations
estimated in the former way are not as easy to use as the latter one. Our
previous work uses a probabilistic method to alleviate the overfitting, but it
is not robust with a small corpus. In this paper, we propose a new neural
network to estimate distributed word representations using a lexicon and a
corpus. We add a lexicon layer in the continuous bag-of-words model and a
threshold node after the output of the lexicon layer. The threshold rejects the
unreliable outputs of the lexicon layer that are less likely to be the same
with their inputs. In this way, it alleviates the overfitting of the polysemous
words. The proposed neural network can be trained using negative sampling,
which maximizing the log probabilities of target words given the context words,
by distinguishing the target words from random noises. We compare the proposed
neural network with the continuous bag-of-words model, the other works
improving it, and the previous works estimating distributed word
representations using both a lexicon and a corpus. The experimental results
show that the proposed neural network is more efficient and balanced for both
semantic tasks and syntactic tasks than the previous works, and robust to the
size of the corpus.

negative:
Words in natural language follow a Zipfian distribution whereby some words
are frequent but most are rare. Learning representations for words in the "long
tail" of this distribution requires enormous amounts of data. Representations
of rare words trained directly on end tasks are usually poor, requiring us to
pre-train embeddings on external data, or treat all rare words as
out-of-vocabulary words with a unique representation. We provide a method for
predicting embeddings of rare words on the fly from small amounts of auxiliary
data with a network trained end-to-end for the downstream task. We show that
this improves results against baselines where embeddings are trained on the end
task for reading comprehension, recognizing textual entailment and language
modeling.

negative:
Loading models pre-trained on the large-scale corpus in the general domain
and fine-tuning them on specific downstream tasks is gradually becoming a
paradigm in Natural Language Processing. Previous investigations prove that
introducing a further pre-training phase between pre-training and fine-tuning
phases to adapt the model on the domain-specific unlabeled data can bring
positive effects. However, most of these further pre-training works just keep
running the conventional pre-training task, e.g., masked language model, which
can be regarded as the domain adaptation to bridge the data distribution gap.
After observing diverse downstream tasks, we suggest that different tasks may
also need a further pre-training phase with appropriate training tasks to
bridge the task formulation gap. To investigate this, we carry out a study for
improving multiple task-oriented dialogue downstream tasks through designing
various tasks at the further pre-training phase. The experiment shows that
different downstream tasks prefer different further pre-training tasks, which
have intrinsic correlation and most further pre-training tasks significantly
improve certain target tasks rather than all. Our investigation indicates that
it is of great importance and effectiveness to design appropriate further
pre-training tasks modeling specific information that benefit downstream tasks.
Besides, we present multiple constructive empirical conclusions for enhancing
task-oriented dialogues.

negative:
We propose an end-to-end neural model for zero-shot abstractive text
summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on
ROCStories, a subset for which we collected human summaries. In this task,
five-sentence stories (paragraphs) are summarized with one sentence, using
human summaries only for evaluation. We show results for extractive and human
baselines to demonstrate a large abstractive gap in performance. Our model,
SummAE, consists of a denoising auto-encoder that embeds sentences and
paragraphs in a common space, from which either can be decoded. Summaries for
paragraphs are generated by decoding a sentence from the paragraph
representations. We find that traditional sequence-to-sequence auto-encoders
fail to produce good summaries and describe how specific architectural choices
and pre-training techniques can significantly improve performance,
outperforming extractive baselines. The data, training, evaluation code, and
best model weights are open-sourced.

negative:
We describe an algorithm for automatic classification of idiomatic and
literal expressions. Our starting point is that words in a given text segment,
such as a paragraph, that are highranking representatives of a common topic of
discussion are less likely to be a part of an idiomatic expression. Our
additional hypothesis is that contexts in which idioms occur, typically, are
more affective and therefore, we incorporate a simple analysis of the intensity
of the emotions expressed by the contexts. We investigate the bag of words
topic representation of one to three paragraphs containing an expression that
should be classified as idiomatic or literal (a target phrase). We extract
topics from paragraphs containing idioms and from paragraphs containing
literals using an unsupervised clustering method, Latent Dirichlet Allocation
(LDA) (Blei et al., 2003). Since idiomatic expressions exhibit the property of
non-compositionality, we assume that they usually present different semantics
than the words used in the local topic. We treat idioms as semantic outliers,
and the identification of a semantic shift as outlier detection. Thus, this
topic representation allows us to differentiate idioms from literals using
local semantic contexts. Our results are encouraging.

negative:
Language models (LMs) pretrained on a large text corpus and fine-tuned on a
downstream text corpus and fine-tuned on a downstream task becomes a de facto
training strategy for several natural language processing (NLP) tasks.
Recently, an adaptive pretraining method retraining the pretrained language
model with task-relevant data has shown significant performance improvements.
However, current adaptive pretraining methods suffer from underfitting on the
task distribution owing to a relatively small amount of data to re-pretrain the
LM. To completely use the concept of adaptive pretraining, we propose a
back-translated task-adaptive pretraining (BT-TAPT) method that increases the
amount of task-specific data for LM re-pretraining by augmenting the task data
using back-translation to generalize the LM to the target task domain. The
experimental results show that the proposed BT-TAPT yields improved
classification accuracy on both low- and high-resource data and better
robustness to noise than the conventional adaptive pretraining method.

negative:
Deep learning currently dominates the benchmarks for various NLP tasks and,
at the basis of such systems, words are frequently represented as embeddings
--vectors in a low dimensional space-- learned from large text corpora and
various algorithms have been proposed to learn both word and concept
embeddings. One of the claimed benefits of such embeddings is that they capture
knowledge about semantic relations. Such embeddings are most often evaluated
through tasks such as predicting human-rated similarity and analogy which only
test a few, often ill-defined, relations. In this paper, we propose a method
for (i) reliably generating word and concept pair datasets for a wide number of
relations by using a knowledge graph and (ii) evaluating to what extent
pre-trained embeddings capture those relations. We evaluate the approach
against a proprietary and a public knowledge graph and analyze the results,
showing which lexico-semantic relational knowledge is captured by current
embedding learning approaches.

negative:
We introduce semantic form mid-tuning, an approach for transferring semantic
knowledge from semantic meaning representations into transformer-based language
encoders. In mid-tuning, we learn to align the text of general sentences -- not
tied to any particular inference task -- and structured semantic
representations of those sentences. Our approach does not require gold
annotated semantic representations. Instead, it makes use of automatically
generated semantic representations, such as from off-the-shelf PropBank and
FrameNet semantic parsers. We show that this alignment can be learned
implicitly via classification or directly via triplet loss. Our method yields
language encoders that demonstrate improved predictive performance across
inference, reading comprehension, textual similarity, and other semantic tasks
drawn from the GLUE, SuperGLUE, and SentEval benchmarks. We evaluate our
approach on three popular baseline models, where our experimental results and
analysis concludes that current pre-trained language models can further benefit
from structured semantic frames with the proposed mid-tuning method, as they
inject additional task-agnostic knowledge to the encoder, improving the
generated embeddings as well as the linguistic properties of the given model,
as evident from improvements on a popular sentence embedding toolkit and a
variety of probing tasks.

negative:
The trends of open science have enabled several open scholarly datasets which
include millions of papers and authors. Managing, exploring, and utilizing such
large and complicated datasets effectively are challenging. In recent years,
the knowledge graph has emerged as a universal data format for representing
knowledge about heterogeneous entities and their relationships. The knowledge
graph can be modeled by knowledge graph embedding methods, which represent
entities and relations as embedding vectors in semantic space, then model the
interactions between these embedding vectors. However, the semantic structures
in the knowledge graph embedding space are not well-studied, thus knowledge
graph embedding methods are usually only used for knowledge graph completion
but not data representation and analysis. In this paper, we propose to analyze
these semantic structures based on the well-studied word embedding space and
use them to support data exploration. We also define the semantic queries,
which are algebraic operations between the embedding vectors in the knowledge
graph embedding space, to solve queries such as similarity and analogy between
the entities on the original datasets. We then design a general framework for
data exploration by semantic queries and discuss the solution to some
traditional scholarly data exploration tasks. We also propose some new
interesting tasks that can be solved based on the uncanny semantic structures
of the embedding space.

negative:
Neural network pretraining is gaining attention due to its outstanding
performance in natural language processing applications. However, pretraining
usually leverages predefined task sequences to learn general linguistic clues.
The lack of mechanisms in choosing proper tasks during pretraining makes the
learning and knowledge encoding inefficient. We thus propose using
meta-learning to select tasks that provide the most informative learning
signals in each episode of pretraining. With the proposed method, we aim to
achieve better efficiency in computation and memory usage for the pretraining
process and resulting networks while maintaining the performance. In this
preliminary work, we discuss the algorithm of the method and its two variants,
downstream-aware and downstream-agnostic pretraining. Our experiment plan is
also summarized, while empirical results will be shared in our future works.

negative:
Word embeddings typically represent different meanings of a word in a single
conflated vector. Empirical analysis of embeddings of ambiguous words is
currently limited by the small size of manually annotated resources and by the
fact that word senses are treated as unrelated individual concepts. We present
a large dataset based on manual Wikipedia annotations and word senses, where
word senses from different words are related by semantic classes. This is the
basis for novel diagnostic tests for an embedding's content: we probe word
embeddings for semantic classes and analyze the embedding space by classifying
embeddings into semantic classes. Our main findings are: (i) Information about
a sense is generally represented well in a single-vector embedding - if the
sense is frequent. (ii) A classifier can accurately predict whether a word is
single-sense or multi-sense, based only on its embedding. (iii) Although rare
senses are not well represented in single-vector embeddings, this does not have
negative impact on an NLP application whose performance depends on frequent
senses.

negative:
Semantic measures are widely used today to estimate the strength of the
semantic relationship between elements of various types: units of language
(e.g., words, sentences, documents), concepts or even instances semantically
characterized (e.g., diseases, genes, geographical locations). Semantic
measures play an important role to compare such elements according to semantic
proxies: texts and knowledge representations, which support their meaning or
describe their nature. Semantic measures are therefore essential for designing
intelligent agents which will for example take advantage of semantic analysis
to mimic human ability to compare abstract or concrete objects. This paper
proposes a comprehensive survey of the broad notion of semantic measure for the
comparison of units of language, concepts or instances based on semantic proxy
analyses. Semantic measures generalize the well-known notions of semantic
similarity, semantic relatedness and semantic distance, which have been
extensively studied by various communities over the last decades (e.g.,
Cognitive Sciences, Linguistics, and Artificial Intelligence to mention a few).

negative:
The manifold hypothesis suggests that word vectors live on a submanifold
within their ambient vector space. We argue that we should, more accurately,
expect them to live on a pinched manifold: a singular quotient of a manifold
obtained by identifying some of its points. The identified, singular points
correspond to polysemous words, i.e. words with multiple meanings. Our point of
view suggests that monosemous and polysemous words can be distinguished based
on the topology of their neighbourhoods. We present two kinds of empirical
evidence to support this point of view: (1) We introduce a topological measure
of polysemy based on persistent homology that correlates well with the actual
number of meanings of a word. (2) We propose a simple, topologically motivated
solution to the SemEval-2010 task on Word Sense Induction & Disambiguation that
produces competitive results.

negative:
Word embeddings, i.e., low-dimensional vector representations such as GloVe
and SGNS, encode word "meaning" in the sense that distances between words'
vectors correspond to their semantic proximity. This enables transfer learning
of semantics for a variety of natural language processing tasks.
  Word embeddings are typically trained on large public corpora such as
Wikipedia or Twitter. We demonstrate that an attacker who can modify the corpus
on which the embedding is trained can control the "meaning" of new and existing
words by changing their locations in the embedding space. We develop an
explicit expression over corpus features that serves as a proxy for distance
between words and establish a causative relationship between its values and
embedding distances. We then show how to use this relationship for two
adversarial objectives: (1) make a word a top-ranked neighbor of another word,
and (2) move a word from one semantic cluster to another.
  An attack on the embedding can affect diverse downstream tasks, demonstrating
for the first time the power of data poisoning in transfer learning scenarios.
We use this attack to manipulate query expansion in information retrieval
systems such as resume search, make certain names more or less visible to named
entity recognition models, and cause new words to be translated to a particular
target word regardless of the language. Finally, we show how the attacker can
generate linguistically likely corpus modifications, thus fooling defenses that
attempt to filter implausible sentences from the corpus using a language model.

negative:
We have developed a method for extracting the coherence features from a
paragraph by matching similar words in its sentences. We conducted an
experiment with a parallel German corpus containing 2000 human-created and 2000
machine-translated paragraphs. The result showed that our method achieved the
best performance (accuracy = 72.3%, equal error rate = 29.8%) when it is
compared with previous methods on various computer-generated text including
translation and paper generation (best accuracy = 67.9%, equal error rate =
32.0%). Experiments on Dutch, another rich resource language, and a low
resource one (Japanese) attained similar performances. It demonstrated the
efficiency of the coherence features at distinguishing computer-translated from
human-created paragraphs on diverse languages.

negative:
The offset method for solving word analogies has become a standard evaluation
tool for vector-space semantic models: it is considered desirable for a space
to represent semantic relations as consistent vector offsets. We show that the
method's reliance on cosine similarity conflates offset consistency with
largely irrelevant neighborhood structure, and propose simple baselines that
should be used to improve the utility of the method in vector space evaluation.

negative:
We present a method for coarse-grained cross-lingual alignment of comparable
texts: segments consisting of contiguous paragraphs that discuss the same theme
(e.g. history, economy) are aligned based on induced multilingual topics. The
method combines three ideas: a two-level LDA model that filters out words that
do not convey themes, an HMM that models the ordering of themes in the
collection of documents, and language-independent concept annotations to serve
as a cross-language bridge and to strengthen the connection between paragraphs
in the same segment through concept relations. The method is evaluated on
English and French data previously used for monolingual alignment. The results
show state-of-the-art performance in both monolingual and cross-lingual
settings.

negative:
In this paper we show that if we want to obtain human evidence about
conventionalization of some phrases, we should ask native speakers about
associations they have to a given phrase and its component words. We have shown
that if component words of a phrase have each other as frequent associations,
then this phrase can be considered as conventionalized. Another type of
conventionalized phrases can be revealed using two factors: low entropy of
phrase associations and low intersection of component word and phrase
associations. The association experiments were performed for the Russian
language.

negative:
Biased language commonly occurs around topics which are of controversial
nature, thus, stirring disagreement between the different involved parties of a
discussion. This is due to the fact that for language and its use,
specifically, the understanding and use of phrases, the stances are cohesive
within the particular groups. However, such cohesiveness does not hold across
groups.
  In collaborative environments or environments where impartial language is
desired (e.g. Wikipedia, news media), statements and the language therein
should represent equally the involved parties and be neutrally phrased. Biased
language is introduced through the presence of inflammatory words or phrases,
or statements that may be incorrect or one-sided, thus violating such
consensus.
  In this work, we focus on the specific case of phrasing bias, which may be
introduced through specific inflammatory words or phrases in a statement. For
this purpose, we propose an approach that relies on a recurrent neural networks
in order to capture the inter-dependencies between words in a phrase that
introduced bias.
  We perform a thorough experimental evaluation, where we show the advantages
of a neural based approach over competitors that rely on word lexicons and
other hand-crafted features in detecting biased language. We are able to
distinguish biased statements with a precision of P=0.92, thus significantly
outperforming baseline models with an improvement of over 30%. Finally, we
release the largest corpus of statements annotated for biased language.

negative:
Inspired by the success of BERT, several multimodal representation learning
approaches have been proposed that jointly represent image and text. These
approaches achieve superior performance by capturing high-level semantic
information from large-scale multimodal pretraining. In particular, LXMERT and
UNITER adopt visual region feature regression and label classification as
pretext tasks. However, they tend to suffer from the problems of noisy labels
and sparse semantic annotations, based on the visual features having been
pretrained on a crowdsourced dataset with limited and inconsistent semantic
labeling. To overcome these issues, we propose unbiased Dense Contrastive
Visual-Linguistic Pretraining (DCVLP), which replaces the region regression and
classification with cross-modality region contrastive learning that requires no
annotations. Two data augmentation strategies (Mask Perturbation and
Intra-/Inter-Adversarial Perturbation) are developed to improve the quality of
negative samples used in contrastive learning. Overall, DCVLP allows
cross-modality dense region contrastive learning in a self-supervised setting
independent of any object annotations. We compare our method against prior
visual-linguistic pretraining frameworks to validate the superiority of dense
contrastive learning on multimodal representation learning.

negative:
Automatic question generation is an important problem in natural language
processing. In this paper we propose a novel adaptive copying recurrent neural
network model to tackle the problem of question generation from sentences and
paragraphs. The proposed model adds a copying mechanism component onto a
bidirectional LSTM architecture to generate more suitable questions adaptively
from the input data. Our experimental results show the proposed model can
outperform the state-of-the-art question generation methods in terms of BLEU
and ROUGE evaluation scores.

negative:
Measuring what linguistic information is encoded in neural models of language
has become popular in NLP. Researchers approach this enterprise by training
"probes" - supervised models designed to extract linguistic structure from
another model's output. One such probe is the structural probe (Hewitt and
Manning, 2019), designed to quantify the extent to which syntactic information
is encoded in contextualised word representations. The structural probe has a
novel design, unattested in the parsing literature, the precise benefit of
which is not immediately obvious. To explore whether syntactic probes would do
better to make use of existing techniques, we compare the structural probe to a
more traditional parser with an identical lightweight parameterisation. The
parser outperforms structural probe on UUAS in seven of nine analysed
languages, often by a substantial amount (e.g. by 11.1 points in English).
Under a second less common metric, however, there is the opposite trend - the
structural probe outperforms the parser. This begs the question: which metric
should we prefer?

negative:
Automatic question generation from paragraphs is an important and challenging
problem, particularly due to the long context from paragraphs. In this paper,
we propose and study two hierarchical models for the task of question
generation from paragraphs. Specifically, we propose (a) a novel hierarchical
BiLSTM model with selective attention and (b) a novel hierarchical Transformer
architecture, both of which learn hierarchical representations of paragraphs.
We model a paragraph in terms of its constituent sentences, and a sentence in
terms of its constituent words. While the introduction of the attention
mechanism benefits the hierarchical BiLSTM model, the hierarchical Transformer,
with its inherent attention and positional encoding mechanisms also performs
better than flat transformer model. We conducted empirical evaluation on the
widely used SQuAD and MS MARCO datasets using standard metrics. The results
demonstrate the overall effectiveness of the hierarchical models over their
flat counterparts. Qualitatively, our hierarchical models are able to generate
fluent and relevant questions

negative:
The following paper presents a method of comparing two sets of vectors. The
method can be applied in all tasks, where it is necessary to measure the
closeness of two objects presented as sets of vectors. It may be applicable
when we compare the meanings of two sentences as part of the problem of
paraphrasing. This is the problem of measuring semantic similarity of two
sentences (group of words). The existing methods are not sensible for the word
order or syntactic connections in the considered sentences. The method appears
to be advantageous because it neither presents a group of words as one scalar
value, nor does it try to show the closeness through an aggregation vector,
which is mean for the set of vectors. Instead of that we measure the cosine of
the angle as the mean for the first group vectors projections (the context) on
one side and each vector of the second group on the other side. The similarity
of two sentences defined by these means does not lose any semantic
characteristics and takes account of the words traits. The method was verified
on the comparison of sentence pairs in Russian.

negative:
We apply entropy agglomeration (EA), a recently introduced algorithm, to
cluster the words of a literary text. EA is a greedy agglomerative procedure
that minimizes projection entropy (PE), a function that can quantify the
segmentedness of an element set. To apply it, the text is reduced to a feature
allocation, a combinatorial object to represent the word occurences in the
text's paragraphs. The experiment results demonstrate that EA, despite its
reduction and simplicity, is useful in capturing significant relationships
among the words in the text. This procedure was implemented in Python and
published as a free software: REBUS.

negative:
Multi-hop reading comprehension (RC) questions are challenging because they
require reading and reasoning over multiple paragraphs. We argue that it can be
difficult to construct large multi-hop RC datasets. For example, even highly
compositional questions can be answered with a single hop if they target
specific entity types, or the facts needed to answer them are redundant. Our
analysis is centered on HotpotQA, where we show that single-hop reasoning can
solve much more of the dataset than previously thought. We introduce a
single-hop BERT-based RC model that achieves 67 F1---comparable to
state-of-the-art multi-hop models. We also design an evaluation setting where
humans are not shown all of the necessary paragraphs for the intended multi-hop
reasoning but can still answer over 80% of questions. Together with detailed
error analysis, these results suggest there should be an increasing focus on
the role of evidence in multi-hop reasoning and possibly even a shift towards
information retrieval style evaluations with large and diverse evidence
collections.

negative:
Text-to-speech systems are typically evaluated on single sentences. When
long-form content, such as data consisting of full paragraphs or dialogues is
considered, evaluating sentences in isolation is not always appropriate as the
context in which the sentences are synthesized is missing. In this paper, we
investigate three different ways of evaluating the naturalness of long-form
text-to-speech synthesis. We compare the results obtained from evaluating
sentences in isolation, evaluating whole paragraphs of speech, and presenting a
selection of speech or text as context and evaluating the subsequent speech. We
find that, even though these three evaluations are based upon the same
material, the outcomes differ per setting, and moreover that these outcomes do
not necessarily correlate with each other. We show that our findings are
consistent between a single speaker setting of read paragraphs and a
two-speaker dialogue scenario. We conclude that to evaluate the quality of
long-form speech, the traditional way of evaluating sentences in isolation does
not suffice, and that multiple evaluations are required.

negative:
We consider the problem of adapting neural paragraph-level question answering
models to the case where entire documents are given as input. Our proposed
solution trains models to produce well calibrated confidence scores for their
results on individual paragraphs. We sample multiple paragraphs from the
documents during training, and use a shared-normalization training objective
that encourages the model to produce globally correct output. We combine this
method with a state-of-the-art pipeline for training models on document QA
data. Experiments demonstrate strong performance on several document QA
datasets. Overall, we are able to achieve a score of 71.3 F1 on the web portion
of TriviaQA, a large improvement from the 56.7 F1 of the previous best system.

negative:
Generating syntactically and semantically valid and relevant questions from
paragraphs is useful with many applications. Manual generation is a
labour-intensive task, as it requires the reading, parsing and understanding of
long passages of text. A number of question generation models based on
sequence-to-sequence techniques have recently been proposed. Most of them
generate questions from sentences only, and none of them is publicly available
as an easy-to-use service. In this paper, we demonstrate ParaQG, a Web-based
system for generating questions from sentences and paragraphs. ParaQG
incorporates a number of novel functionalities to make the question generation
process user-friendly. It provides an interactive interface for a user to
select answers with visual insights on generation of questions. It also employs
various faceted views to group similar questions as well as filtering
techniques to eliminate unanswerable questions

negative:
One key consequence of the information revolution is a significant increase
and a contamination of our information supply. The practice of fact checking
won't suffice to eliminate the biases in text data we observe, as the degree of
factuality alone does not determine whether biases exist in the spectrum of
opinions visible to us. To better understand controversial issues, one needs to
view them from a diverse yet comprehensive set of perspectives. For example,
there are many ways to respond to a claim such as "animals should have lawful
rights", and these responses form a spectrum of perspectives, each with a
stance relative to this claim and, ideally, with evidence supporting it.
Inherently, this is a natural language understanding task, and we propose to
address it as such. Specifically, we propose the task of substantiated
perspective discovery where, given a claim, a system is expected to discover a
diverse set of well-corroborated perspectives that take a stance with respect
to the claim. Each perspective should be substantiated by evidence paragraphs
which summarize pertinent results and facts. We construct PERSPECTRUM, a
dataset of claims, perspectives and evidence, making use of online debate
websites to create the initial data collection, and augmenting it using search
engines in order to expand and diversify our dataset. We use crowd-sourcing to
filter out noise and ensure high-quality data. Our dataset contains 1k claims,
accompanied with pools of 10k and 8k perspective sentences and evidence
paragraphs, respectively. We provide a thorough analysis of the dataset to
highlight key underlying language understanding challenges, and show that human
baselines across multiple subtasks far outperform ma-chine baselines built upon
state-of-the-art NLP techniques. This poses a challenge and opportunity for the
NLP community to address.

negative:
Children learn word meanings by tapping into the commonalities across
different situations in which words are used and overcome the high level of
uncertainty involved in early word learning experiences. We propose a modeling
framework to investigate the role of mutual exclusivity bias - asserting
one-to-one mappings between words and their meanings - in reducing uncertainty
in word learning. In a set of computational studies, we show that to
successfully learn word meanings in the face of uncertainty, a learner needs to
use two types of competition: words competing for association to a referent
when learning from an observation and referents competing for a word when the
word is used. Our work highlights the importance of an algorithmic-level
analysis to shed light on the utility of different mechanisms that can
implement the same computational-level theory.

negative:
This paper introduces a new framework for open-domain question answering in
which the retriever and the reader iteratively interact with each other. The
framework is agnostic to the architecture of the machine reading model, only
requiring access to the token-level hidden representations of the reader. The
retriever uses fast nearest neighbor search to scale to corpora containing
millions of paragraphs. A gated recurrent unit updates the query at each step
conditioned on the state of the reader and the reformulated query is used to
re-rank the paragraphs by the retriever. We conduct analysis and show that
iterative interaction helps in retrieving informative paragraphs from the
corpus. Finally, we show that our multi-step-reasoning framework brings
consistent improvement when applied to two widely used reader architectures
DrQA and BiDAF on various large open-domain datasets --- TriviaQA-unfiltered,
QuasarT, SearchQA, and SQuAD-Open.

negative:
We study the time taken by a language learner to correctly identify the
meaning of all words in a lexicon under conditions where many plausible
meanings can be inferred whenever a word is uttered. We show that the most
basic form of cross-situational learning - whereby information from multiple
episodes is combined to eliminate incorrect meanings - can perform badly when
words are learned independently and meanings are drawn from a nonuniform
distribution. If learners further assume that no two words share a common
meaning, we find a phase transition between a maximally-efficient learning
regime, where the learning time is reduced to the shortest it can possibly be,
and a partially-efficient regime where incorrect candidate meanings for words
persist at late times. We obtain exact results for the word-learning process
through an equivalence to a statistical mechanical problem of enumerating loops
in the space of word-meaning mappings.