query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : As a linguist, I aim to efficiently categorize various semantic elements within a sentence. For instance, I want to pinpoint the predicate-argument structure. I plan to create a model based on neural networks that enhances traditional methods which utilize syntactic trees. Rather than directly employing syntactic trees, my model will learn to convert syntax and grammatical structure into vectorized representations. These hidden representations will allow my model to be differentiable and converge rapidly. I'm also curious to see if using a semantic representation of my text from a large pre-trained language model could boost the accuracy of my labeling. Ultimately, I intend to test my model against a wide range of linguistic benchmarks.

positive:
Semantic role labeling (SRL) is dedicated to recognizing the semantic
predicate-argument structure of a sentence. Previous studies in terms of
traditional models have shown syntactic information can make remarkable
contributions to SRL performance; however, the necessity of syntactic
information was challenged by a few recent neural SRL studies that demonstrate
impressive performance without syntactic backbones and suggest that syntax
information becomes much less important for neural semantic role labeling,
especially when paired with recent deep neural network and large-scale
pre-trained language models. Despite this notion, the neural SRL field still
lacks a systematic and full investigation on the relevance of syntactic
information in SRL, for both dependency and both monolingual and multilingual
settings. This paper intends to quantify the importance of syntactic
information for neural SRL in the deep learning framework. We introduce three
typical SRL frameworks (baselines), sequence-based, tree-based, and
graph-based, which are accompanied by two categories of exploiting syntactic
information: syntax pruning-based and syntax feature-based. Experiments are
conducted on the CoNLL-2005, 2009, and 2012 benchmarks for all languages
available, and results show that neural SRL models can still benefit from
syntactic information under certain conditions. Furthermore, we show the
quantitative significance of syntax to neural SRL models together with a
thorough empirical survey using existing models.

positive:
Semantic role labeling (SRL), also known as shallow semantic parsing, is an
important yet challenging task in NLP. Motivated by the close correlation
between syntactic and semantic structures, traditional discrete-feature-based
SRL approaches make heavy use of syntactic features. In contrast,
deep-neural-network-based approaches usually encode the input sentence as a
word sequence without considering the syntactic structures. In this work, we
investigate several previous approaches for encoding syntactic trees, and make
a thorough study on whether extra syntax-aware representations are beneficial
for neural SRL models. Experiments on the benchmark CoNLL-2005 dataset show
that syntax-aware SRL approaches can effectively improve performance over a
strong baseline with external word representations from ELMo. With the extra
syntax-aware representations, our approaches achieve new state-of-the-art 85.6
F1 (single model) and 86.6 F1 (ensemble) on the test data, outperforming the
corresponding strong baselines with ELMo by 0.8 and 1.0, respectively. Detailed
error analysis are conducted to gain more insights on the investigated
approaches.

positive:
Current state-of-the-art semantic role labeling (SRL) uses a deep neural
network with no explicit linguistic features. However, prior work has shown
that gold syntax trees can dramatically improve SRL decoding, suggesting the
possibility of increased accuracy from explicit modeling of syntax. In this
work, we present linguistically-informed self-attention (LISA): a neural
network model that combines multi-head self-attention with multi-task learning
across dependency parsing, part-of-speech tagging, predicate detection and SRL.
Unlike previous models which require significant pre-processing to prepare
linguistic features, LISA can incorporate syntax using merely raw tokens as
input, encoding the sequence only once to simultaneously perform parsing,
predicate detection and role labeling for all predicates. Syntax is
incorporated by training one attention head to attend to syntactic parents for
each token. Moreover, if a high-quality syntactic parse is already available,
it can be beneficially injected at test time without re-training our SRL model.
In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art
performance for a model using predicted predicates and standard word
embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art
on newswire and more than 3.5 F1 on out-of-domain data, nearly 10% reduction in
error. On ConLL-2012 English SRL we also show an improvement of more than 2.5
F1. LISA also out-performs the state-of-the-art with contextually-encoded
(ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on
out-of-domain text.

positive:
The addition of syntax-aware decoding in Neural Machine Translation (NMT)
systems requires an effective tree-structured neural network, a syntax-aware
attention model and a language generation model that is sensitive to sentence
structure. We exploit a top-down tree-structured model called DRNN
(Doubly-Recurrent Neural Networks) first proposed by Alvarez-Melis and Jaakola
(2017) to create an NMT model called Seq2DRNN that combines a sequential
encoder with tree-structured decoding augmented with a syntax-aware attention
model. Unlike previous approaches to syntax-based NMT which use dependency
parsing models our method uses constituency parsing which we argue provides
useful information for translation. In addition, we use the syntactic structure
of the sentence to add new connections to the tree-structured decoder neural
network (Seq2DRNN+SynC). We compare our NMT model with sequential and state of
the art syntax-based NMT models and show that our model produces more fluent
translations with better reordering. Since our model is capable of doing
translation and constituency parsing at the same time we also compare our
parsing accuracy against other neural parsing models.

positive:
For natural language processing systems, two kinds of evidence support the
use of text representations from neural language models "pretrained" on large
unannotated corpora: performance on application-inspired benchmarks (Peters et
al., 2018, inter alia), and the emergence of syntactic abstractions in those
representations (Tenney et al., 2019, inter alia). On the other hand, the lack
of grounded supervision calls into question how well these representations can
ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent
language models -- specifically focusing on predicate-argument structure as
operationalized by semantic dependencies (Ivanova et al., 2012) -- and find
that, unlike syntax, semantics is not brought to the surface by today's
pretrained models. We then use convolutional graph encoders to explicitly
incorporate semantic parses into task-specific finetuning, yielding benefits to
natural language understanding (NLU) tasks in the GLUE benchmark. This approach
demonstrates the potential for general-purpose (rather than task-specific)
linguistic supervision, above and beyond conventional pretraining and
finetuning. Several diagnostics help to localize the benefits of our approach.

positive:
Many natural language understanding (NLU) tasks, such as shallow parsing
(i.e., text chunking) and semantic slot filling, require the assignment of
representative labels to the meaningful chunks in a sentence. Most of the
current deep neural network (DNN) based methods consider these tasks as a
sequence labeling problem, in which a word, rather than a chunk, is treated as
the basic unit for labeling. These chunks are then inferred by the standard IOB
(Inside-Outside-Beginning) labels. In this paper, we propose an alternative
approach by investigating the use of DNN for sequence chunking, and propose
three neural models so that each chunk can be treated as a complete unit for
labeling. Experimental results show that the proposed neural sequence chunking
models can achieve start-of-the-art performance on both the text chunking and
slot filling tasks.

positive:
Tree-based Long short term memory (LSTM) network has become state-of-the-art
for modeling the meaning of language texts as they can effectively exploit the
grammatical syntax and thereby non-linear dependencies among words of the
sentence. However, most of these models cannot recognize the difference in
meaning caused by a change in semantic roles of words or phrases because they
do not acknowledge the type of grammatical relations, also known as typed
dependencies, in sentence structure. This paper proposes an enhanced LSTM
architecture, called relation gated LSTM, which can model the relationship
between two inputs of a sequence using a control input. We also introduce a
Tree-LSTM model called Typed Dependency Tree-LSTM that uses the sentence
dependency parse structure as well as the dependency type to embed sentence
meaning into a dense vector. The proposed model outperformed its type-unaware
counterpart in two typical NLP tasks - Semantic Relatedness Scoring and
Sentiment Analysis, in a lesser number of training epochs. The results were
comparable or competitive with other state-of-the-art models. Qualitative
analysis showed that changes in the voice of sentences had little effect on the
model's predicted scores, while changes in nominal (noun) words had a more
significant impact. The model recognized subtle semantic relationships in
sentence pairs. The magnitudes of learned typed dependencies embeddings were
also in agreement with human intuitions. The research findings imply the
significance of grammatical relations in sentence modeling. The proposed models
would serve as a base for future researches in this direction.

positive:
Deep neural networks for machine comprehension typically utilizes only word
or character embeddings without explicitly taking advantage of structured
linguistic information such as constituency trees and dependency trees. In this
paper, we propose structural embedding of syntactic trees (SEST), an algorithm
framework to utilize structured information and encode them into vector
representations that can boost the performance of algorithms for the machine
comprehension. We evaluate our approach using a state-of-the-art neural
attention model on the SQuAD dataset. Experimental results demonstrate that our
model can accurately identify the syntactic boundaries of the sentences and
extract answers that are syntactically coherent over the baseline methods.

positive:
Progress in pre-trained language models has led to a surge of impressive
results on downstream tasks for natural language understanding. Recent work on
probing pre-trained language models uncovered a wide range of linguistic
properties encoded in their contextualized representations. However, it is
unclear whether they encode semantic knowledge that is crucial to symbolic
inference methods. We propose a methodology for probing linguistic information
for logical inference in pre-trained language model representations. Our
probing datasets cover a list of linguistic phenomena required by major
symbolic inference systems. We find that (i) pre-trained language models do
encode several types of linguistic information for inference, but there are
also some types of information that are weakly encoded, (ii) language models
can effectively learn missing linguistic information through fine-tuning.
Overall, our findings provide insights into which aspects of linguistic
information for logical inference do language models and their pre-training
procedures capture. Moreover, we have demonstrated language models' potential
as semantic and background knowledge bases for supporting symbolic inference
methods.

positive:
Due to the increasing amount of data on the internet, finding a
highly-informative, low-dimensional representation for text is one of the main
challenges for efficient natural language processing tasks including text
classification. This representation should capture the semantic information of
the text while retaining their relevance level for document classification.
This approach maps the documents with similar topics to a similar space in
vector space representation. To obtain representation for large text, we
propose the utilization of deep Siamese neural networks. To embed document
relevance in topics in the distributed representation, we use a Siamese neural
network to jointly learn document representations. Our Siamese network consists
of two sub-network of multi-layer perceptron. We examine our representation for
the text categorization task on BBC news dataset. The results show that the
proposed representations outperform the conventional and state-of-the-art
representations in the text classification task on this dataset.

positive:
We propose a generative model for a sentence that uses two latent variables,
with one intended to represent the syntax of the sentence and the other to
represent its semantics. We show we can achieve better disentanglement between
semantic and syntactic representations by training with multiple losses,
including losses that exploit aligned paraphrastic sentences and word-order
information. We also investigate the effect of moving from bag-of-words to
recurrent neural network modules. We evaluate our models as well as several
popular pretrained embeddings on standard semantic similarity tasks and novel
syntactic similarity tasks. Empirically, we find that the model with the best
performing syntactic and semantic representations also gives rise to the most
disentangled representations.

positive:
Neural networks trained on natural language processing tasks capture syntax
even though it is not provided as a supervision signal. This indicates that
syntactic analysis is essential to the understating of language in artificial
intelligence systems. This overview paper covers approaches of evaluating the
amount of syntactic information included in the representations of words for
different neural network architectures. We mainly summarize re-search on
English monolingual data on language modeling tasks and multilingual data for
neural machine translation systems and multilingual language models. We
describe which pre-trained models and representations of language are best
suited for transfer to syntactic tasks.

positive:
We study the problem of leveraging the syntactic structure of text to enhance
pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of
text either in the pre-training stage or in the fine-tuning stage, so that they
suffer from discrepancy between the two stages. Such a problem would lead to
the necessity of having human-annotated syntactic information, which limits the
application of existing methods to broader scenarios. To address this, we
present a model that utilizes the syntax of text in both pre-training and
fine-tuning stages. Our model is based on Transformer with a syntax-aware
attention layer that considers the dependency tree of the text. We further
introduce a new pre-training task of predicting the syntactic distance among
tokens in the dependency tree. We evaluate the model on three downstream tasks,
including relation classification, entity typing, and question answering.
Results show that our model achieves state-of-the-art performance on six public
benchmark datasets. We have two major findings. First, we demonstrate that
infusing automatically produced syntax of text improves pre-trained models.
Second, global syntactic distances among tokens bring larger performance gains
compared to local head relations between contiguous tokens.

positive:
The latest developments in neural semantic role labeling (SRL) have shown
great performance improvements with both the dependency and span
formalisms/styles. Although the two styles share many similarities in
linguistic meaning and computation, most previous studies focus on a single
style. In this paper, we define a new cross-style semantic role label
convention and propose a new cross-style joint optimization model designed
around the most basic linguistic meaning of a semantic role, providing a
solution to make the results of the two styles more comparable and allowing
both formalisms of SRL to benefit from their natural connections in both
linguistics and computation. Our model learns a general semantic argument
structure and is capable of outputting in either style. Additionally, we
propose a syntax-aided method to uniformly enhance the learning of both
dependency and span representations. Experiments show that the proposed methods
are effective on both span and dependency SRL benchmarks.

positive:
Sense representations have gone beyond word representations like Word2Vec,
GloVe and FastText and achieved innovative performance on a wide range of
natural language processing tasks. Although very useful in many applications,
the traditional approaches for generating word embeddings have a strict
drawback: they produce a single vector representation for a given word ignoring
the fact that ambiguous words can assume different meanings. In this paper, we
explore unsupervised sense representations which, different from traditional
word embeddings, are able to induce different senses of a word by analyzing its
contextual semantics in a text. The unsupervised sense representations
investigated in this paper are: sense embeddings and deep neural language
models. We present the first experiments carried out for generating sense
embeddings for Portuguese. Our experiments show that the sense embedding model
(Sense2vec) outperformed traditional word embeddings in syntactic and semantic
analogies task, proving that the language resource generated here can improve
the performance of NLP tasks in Portuguese. We also evaluated the performance
of pre-trained deep neural language models (ELMo and BERT) in two transfer
learning approaches: feature based and fine-tuning, in the semantic textual
similarity task. Our experiments indicate that the fine tuned Multilingual and
Portuguese BERT language models were able to achieve better accuracy than the
ELMo model and baselines.

positive:
Existing neural semantic parsers mainly utilize a sequence encoder, i.e., a
sequential LSTM, to extract word order features while neglecting other valuable
syntactic information such as dependency graph or constituent trees. In this
paper, we first propose to use the \textit{syntactic graph} to represent three
types of syntactic information, i.e., word order, dependency and constituency
features. We further employ a graph-to-sequence model to encode the syntactic
graph and decode a logical form. Experimental results on benchmark datasets
show that our model is comparable to the state-of-the-art on Jobs640, ATIS and
Geo880. Experimental results on adversarial examples demonstrate the robustness
of the model is also improved by encoding more syntactic information.

positive:
Semantic role labeling (SRL) is the task of identifying the
predicate-argument structure of a sentence. It is typically regarded as an
important step in the standard NLP pipeline. As the semantic representations
are closely related to syntactic ones, we exploit syntactic information in our
model. We propose a version of graph convolutional networks (GCNs), a recent
class of neural networks operating on graphs, suited to model syntactic
dependency graphs. GCNs over syntactic dependency trees are used as sentence
encoders, producing latent feature representations of words in a sentence. We
observe that GCN layers are complementary to LSTM ones: when we stack both GCN
and LSTM layers, we obtain a substantial improvement over an already
state-of-the-art LSTM SRL model, resulting in the best reported scores on the
standard benchmark (CoNLL-2009) both for Chinese and English.

positive:
Syntactic structure of sentences in a document substantially informs about
its authorial writing style. Sentence representation learning has been widely
explored in recent years and it has been shown that it improves the
generalization of different downstream tasks across many domains. Even though
utilizing probing methods in several studies suggests that these learned
contextual representations implicitly encode some amount of syntax, explicit
syntactic information further improves the performance of deep neural models in
the domain of authorship attribution. These observations have motivated us to
investigate the explicit representation learning of syntactic structure of
sentences. In this paper, we propose a self-supervised framework for learning
structural representations of sentences. The self-supervised network contains
two components; a lexical sub-network and a syntactic sub-network which take
the sequence of words and their corresponding structural labels as the input,
respectively. Due to the n-to-1 mapping of words to their structural labels,
each word will be embedded into a vector representation which mainly carries
structural information. We evaluate the learned structural representations of
sentences using different probing tasks, and subsequently utilize them in the
authorship attribution task. Our experimental results indicate that the
structural embeddings significantly improve the classification tasks when
concatenated with the existing pre-trained word embeddings.

positive:
Machine translation systems require semantic knowledge and grammatical
understanding. Neural machine translation (NMT) systems often assume this
information is captured by an attention mechanism and a decoder that ensures
fluency. Recent work has shown that incorporating explicit syntax alleviates
the burden of modeling both types of knowledge. However, requiring parses is
expensive and does not explore the question of what syntax a model needs during
translation. To address both of these issues we introduce a model that
simultaneously translates while inducing dependency trees. In this way, we
leverage the benefits of structure while investigating what syntax NMT must
induce to maximize performance. We show that our dependency trees are 1.
language pair dependent and 2. improve translation quality.

positive:
Sentence semantic understanding is a key topic in the field of natural
language processing. Recently, contextualized word representations derived from
pre-trained language models such as ELMO and BERT have shown significant
improvements for a wide range of semantic tasks, e.g. question answering, text
classification and sentiment analysis. However, how to add external knowledge
to further improve the semantic modeling capability of model is worth probing.
In this paper, we propose a novel approach to combining syntax information with
a pre-trained language model. In order to evaluate the effect of the
pre-training model, first, we introduce RNN-based and Transformer-based
pre-trained language models; secondly, to better integrate external knowledge,
such as syntactic information integrate with the pre-training model, we propose
a dependency syntax expansion (DSE) model. For evaluation, we have selected two
subtasks: sentence completion task and biological relation extraction task. The
experimental results show that our model achieves 91.2\% accuracy,
outperforming the baseline model by 37.8\% on sentence completion task. And it
also gets competitive performance by 75.1\% $F_{1}$ score on relation
extraction task.

positive:
While a lot of analysis has been carried to demonstrate linguistic knowledge
captured by the representations learned within deep NLP models, very little
attention has been paid towards individual neurons.We carry outa neuron-level
analysis using core linguistic tasks of predicting morphology, syntax and
semantics, on pre-trained language models, with questions like: i) do
individual neurons in pre-trained models capture linguistic information? ii)
which parts of the network learn more about certain linguistic phenomena? iii)
how distributed or focused is the information? and iv) how do various
architectures differ in learning these properties? We found small subsets of
neurons to predict linguistic tasks, with lower level tasks (such as
morphology) localized in fewer neurons, compared to higher level task of
predicting syntax. Our study also reveals interesting cross architectural
comparisons. For example, we found neurons in XLNet to be more localized and
disjoint when predicting properties compared to BERT and others, where they are
more distributed and coupled.

positive:
Writing style is a combination of consistent decisions at different levels of
language production including lexical, syntactic, and structural associated to
a specific author (or author groups). While lexical-based models have been
widely explored in style-based text classification, relying on content makes
the model less scalable when dealing with heterogeneous data comprised of
various topics. On the other hand, syntactic models which are
content-independent, are more robust against topic variance. In this paper, we
introduce a syntactic recurrent neural network to encode the syntactic patterns
of a document in a hierarchical structure. The model first learns the syntactic
representation of sentences from the sequence of part-of-speech tags. For this
purpose, we exploit both convolutional filters and long short-term memories to
investigate the short-term and long-term dependencies of part-of-speech tags in
the sentences. Subsequently, the syntactic representations of sentences are
aggregated into document representation using recurrent neural networks. Our
experimental results on PAN 2012 dataset for authorship attribution task shows
that syntactic recurrent neural network outperforms the lexical model with the
identical architecture by approximately 14% in terms of accuracy.

positive:
This paper investigates whether the power of the models pre-trained on text
data, such as BERT, can be transferred to general token sequence classification
applications. To verify pre-trained models' transferability, we test the
pre-trained models on text classification tasks with meanings of tokens
mismatches, and real-world non-text token sequence classification data,
including amino acid, DNA, and music. We find that even on non-text data, the
models pre-trained on text converge faster, perform better than the randomly
initialized models, and only slightly worse than the models using task-specific
knowledge. We also find that the representations of the text and non-text
pre-trained models share non-trivial similarities.

positive:
We study the problem of integrating syntactic information from constituency
trees into a neural model in Frame-semantic parsing sub-tasks, namely Target
Identification (TI), FrameIdentification (FI), and Semantic Role Labeling
(SRL). We use a Graph Convolutional Network to learn specific representations
of constituents, such that each constituent is profiled as the production
grammar rule it corresponds to. We leverage these representations to build
syntactic features for each word in a sentence, computed as the sum of all the
constituents on the path between a word and a task-specific node in the tree,
e.g. the target predicate for SRL. Our approach improves state-of-the-art
results on the TI and SRL of ~1%and~3.5% points, respectively (+2.5% additional
points are gained with BERT as input), when tested on FrameNet 1.5, while
yielding comparable results on the CoNLL05 dataset to other syntax-aware
systems.

positive:
We study methods for learning sentence embeddings with syntactic structure.
We focus on methods of learning syntactic sentence-embeddings by using a
multilingual parallel-corpus augmented by Universal Parts-of-Speech tags. We
evaluate the quality of the learned embeddings by examining sentence-level
nearest neighbours and functional dissimilarity in the embedding space. We also
evaluate the ability of the method to learn syntactic sentence-embeddings for
low-resource languages and demonstrate strong evidence for transfer learning.
Our results show that syntactic sentence-embeddings can be learned while using
less training data, fewer model parameters, and resulting in better evaluation
metrics than state-of-the-art language models.

positive:
Neuroscientists evaluate deep neural networks for natural language processing
as possible candidate models for how language is processed in the brain. These
models are often trained without explicit linguistic supervision, but have been
shown to learn some linguistic structure in the absence of such supervision
(Manning et al., 2020), potentially questioning the relevance of symbolic
linguistic theories in modeling such cognitive processes (Warstadt and Bowman,
2020). We evaluate across two fMRI datasets whether language models align
better with brain recordings, if their attention is biased by annotations from
syntactic or semantic formalisms. Using structure from dependency or minimal
recursion semantic annotations, we find alignments improve significantly for
one of the datasets. For another dataset, we see more mixed results. We present
an extensive analysis of these results. Our proposed approach enables the
evaluation of more targeted hypotheses about the composition of meaning in the
brain, expanding the range of possible scientific inferences a neuroscientist
could make, and opens up new opportunities for cross-pollination between
computational neuroscience and linguistics.

positive:
Text classification is one of the most widely studied tasks in natural
language processing. Motivated by the principle of compositionality, large
multilayer neural network models have been employed for this task in an attempt
to effectively utilize the constituent expressions. Almost all of the reported
work train large networks using discriminative approaches, which come with a
caveat of no proper capacity control, as they tend to latch on to any signal
that may not generalize. Using various recent state-of-the-art approaches for
text classification, we explore whether these models actually learn to compose
the meaning of the sentences or still just focus on some keywords or lexicons
for classifying the document. To test our hypothesis, we carefully construct
datasets where the training and test splits have no direct overlap of such
lexicons, but overall language structure would be similar. We study various
text classifiers and observe that there is a big performance drop on these
datasets. Finally, we show that even simple models with our proposed
regularization techniques, which disincentivize focusing on key lexicons, can
substantially improve classification accuracy.

positive:
In natural languages, words are used in association to construct sentences.
It is not words in isolation, but the appropriate combination of hierarchical
structures that conveys the meaning of the whole sentence. Neural networks can
capture expressive language features; however, insights into the link between
words and sentences are difficult to acquire automatically. In this work, we
design a deep neural network architecture that explicitly wires lower and
higher linguistic components; we then evaluate its ability to perform the same
task at different hierarchical levels. Settling on broad text classification
tasks, we show that our model, MHAL, learns to simultaneously solve them at
different levels of granularity by fluidly transferring knowledge between
hierarchies. Using a multi-head attention mechanism to tie the representations
between single words and full sentences, MHAL systematically outperforms
equivalent models that are not incentivized towards developing compositional
representations. Moreover, we demonstrate that, with the proposed architecture,
the sentence information flows naturally to individual words, allowing the
model to behave like a sequence labeller (which is a lower, word-level task)
even without any word supervision, in a zero-shot fashion.

positive:
We propose a neural language model capable of unsupervised syntactic
structure induction. The model leverages the structure information to form
better semantic representations and better language modeling. Standard
recurrent neural networks are limited by their structure and fail to
efficiently use syntactic information. On the other hand, tree-structured
recursive networks usually require additional structural supervision at the
cost of human expert annotation. In this paper, We propose a novel neural
language model, called the Parsing-Reading-Predict Networks (PRPN), that can
simultaneously induce the syntactic structure from unannotated sentences and
leverage the inferred structure to learn a better language model. In our model,
the gradient can be directly back-propagated from the language model loss into
the neural parsing network. Experiments show that the proposed model can
discover the underlying syntactic structure and achieve state-of-the-art
performance on word/character-level language model tasks.

positive:
This paper proposes a neural semantic parsing approach -- Sequence-to-Action,
which models semantic parsing as an end-to-end semantic graph generation
process. Our method simultaneously leverages the advantages from two recent
promising directions of semantic parsing. Firstly, our model uses a semantic
graph to represent the meaning of a sentence, which has a tight-coupling with
knowledge bases. Secondly, by leveraging the powerful representation learning
and prediction ability of neural network models, we propose a RNN model which
can effectively map sentences to action sequences for semantic graph
generation. Experiments show that our method achieves state-of-the-art
performance on OVERNIGHT dataset and gets competitive performance on GEO and
ATIS datasets.

positive:
We present novel methods for analyzing the activation patterns of RNNs from a
linguistic point of view and explore the types of linguistic structure they
learn. As a case study, we use a multi-task gated recurrent network
architecture consisting of two parallel pathways with shared word embeddings
trained on predicting the representations of the visual scene corresponding to
an input sentence, and predicting the next word in the same sentence. Based on
our proposed method to estimate the amount of contribution of individual tokens
in the input to the final prediction of the networks we show that the image
prediction pathway: a) is sensitive to the information structure of the
sentence b) pays selective attention to lexical categories and grammatical
functions that carry semantic information c) learns to treat the same input
token differently depending on its grammatical functions in the sentence. In
contrast the language model is comparatively more sensitive to words with a
syntactic function. Furthermore, we propose methods to ex- plore the function
of individual hidden units in RNNs and show that the two pathways of the
architecture in our case study contain specialized units tuned to patterns
informative for the task, some of which can carry activations to later time
steps to encode long-term dependencies.

positive:
This paper describes a neural semantic parser that maps natural language
utterances onto logical forms which can be executed against a task-specific
environment, such as a knowledge base or a database, to produce a response. The
parser generates tree-structured logical forms with a transition-based approach
which combines a generic tree-generation algorithm with domain-general
operations defined by the logical language. The generation process is modeled
by structured recurrent neural networks, which provide a rich encoding of the
sentential context and generation history for making predictions. To tackle
mismatches between natural language and logical form tokens, various attention
mechanisms are explored. Finally, we consider different training settings for
the neural semantic parser, including a fully supervised training where
annotated logical forms are given, weakly-supervised training where denotations
are provided, and distant supervision where only unlabeled sentences and a
knowledge base are available. Experiments across a wide range of datasets
demonstrate the effectiveness of our parser.

positive:
While state-of-the-art neural network models continue to achieve lower
perplexity scores on language modeling benchmarks, it remains unknown whether
optimizing for broad-coverage predictive performance leads to human-like
syntactic knowledge. Furthermore, existing work has not provided a clear
picture about the model properties required to produce proper syntactic
generalizations. We present a systematic evaluation of the syntactic knowledge
of neural language models, testing 20 combinations of model types and data
sizes on a set of 34 English-language syntactic test suites. We find
substantial differences in syntactic generalization performance by model
architecture, with sequential models underperforming other architectures.
Factorially manipulating model architecture and training dataset size (1M--40M
words), we find that variability in syntactic generalization performance is
substantially greater by architecture than by dataset size for the corpora
tested in our experiments. Our results also reveal a dissociation between
perplexity and syntactic generalization performance.

negative:
We present a set of experiments to demonstrate that deep recurrent neural
networks (RNNs) learn internal representations that capture soft hierarchical
notions of syntax from highly varied supervision. We consider four syntax tasks
at different depths of the parse tree; for each word, we predict its part of
speech as well as the first (parent), second (grandparent) and third level
(great-grandparent) constituent labels that appear above it. These predictions
are made from representations produced at different depths in networks that are
pretrained with one of four objectives: dependency parsing, semantic role
labeling, machine translation, or language modeling. In every case, we find a
correspondence between network depth and syntactic depth, suggesting that a
soft syntactic hierarchy emerges. This effect is robust across all conditions,
indicating that the models encode significant amounts of syntax even in the
absence of an explicit syntactic training supervision.

negative:
Grammatical Error Correction (GEC) is a task of detecting and correcting
grammatical errors in sentences. Recently, neural machine translation systems
have become popular approaches for this task. However, these methods lack the
use of syntactic knowledge which plays an important role in the correction of
grammatical errors. In this work, we propose a syntax-guided GEC model (SG-GEC)
which adopts the graph attention mechanism to utilize the syntactic knowledge
of dependency trees. Considering the dependency trees of the grammatically
incorrect source sentences might provide incorrect syntactic knowledge, we
propose a dependency tree correction task to deal with it. Combining with data
augmentation method, our model achieves strong performances without using any
large pre-trained models. We evaluate our model on public benchmarks of GEC
task and it achieves competitive results.

negative:
Building robust natural language understanding systems will require a clear
characterization of whether and how various linguistic meaning representations
complement each other. To perform a systematic comparative analysis, we
evaluate the mapping between meaning representations from different frameworks
using two complementary methods: (i) a rule-based converter, and (ii) a
supervised delexicalized parser that parses to one framework using only
information from the other as features. We apply these methods to convert the
STREUSLE corpus (with syntactic and lexical semantic annotations) to UCCA (a
graph-structured full-sentence meaning representation). Both methods yield
surprisingly accurate target representations, close to fully supervised UCCA
parser quality---indicating that UCCA annotations are partially redundant with
STREUSLE annotations. Despite this substantial convergence between frameworks,
we find several important areas of divergence.

negative:
The success of long short-term memory (LSTM) neural networks in language
processing is typically attributed to their ability to capture long-distance
statistical regularities. Linguistic regularities are often sensitive to
syntactic structure; can such dependencies be captured by LSTMs, which do not
have explicit structural representations? We begin addressing this question
using number agreement in English subject-verb dependencies. We probe the
architecture's grammatical competence both using training objectives with an
explicit grammatical target (number prediction, grammaticality judgments) and
using language models. In the strongly supervised settings, the LSTM achieved
very high overall accuracy (less than 1% errors), but errors increased when
sequential and structural information conflicted. The frequency of such errors
rose sharply in the language-modeling setting. We conclude that LSTMs can
capture a non-trivial amount of grammatical structure given targeted
supervision, but stronger architectures may be required to further reduce
errors; furthermore, the language modeling signal is insufficient for capturing
syntax-sensitive dependencies, and should be supplemented with more direct
supervision if such dependencies need to be captured.

negative:
We introduce a new test of how well language models capture meaning in
children's books. Unlike standard language modelling benchmarks, it
distinguishes the task of predicting syntactic function words from that of
predicting lower-frequency words, which carry greater semantic content. We
compare a range of state-of-the-art models, each with a different way of
encoding what has been previously read. We show that models which store
explicit representations of long-term contexts outperform state-of-the-art
neural language models at predicting semantic content words, although this
advantage is not observed for syntactic function words. Interestingly, we find
that the amount of text encoded in a single memory representation is highly
influential to the performance: there is a sweet-spot, not too big and not too
small, between single words and full sentences that allows the most meaningful
information in a text to be effectively retained and recalled. Further, the
attention over such window-based memories can be trained effectively through
self-supervision. We then assess the generality of this principle by applying
it to the CNN QA benchmark, which involves identifying named entities in
paraphrased summaries of news articles, and achieve state-of-the-art
performance.

negative:
In this dissertation we report results of our research on dense distributed
representations of text data. We propose two novel neural models for learning
such representations. The first model learns representations at the document
level, while the second model learns word-level representations.
  For document-level representations we propose Binary Paragraph Vector: a
neural network models for learning binary representations of text documents,
which can be used for fast document retrieval. We provide a thorough evaluation
of these models and demonstrate that they outperform the seminal method in the
field in the information retrieval task. We also report strong results in
transfer learning settings, where our models are trained on a generic text
corpus and then used to infer codes for documents from a domain-specific
dataset. In contrast to previously proposed approaches, Binary Paragraph Vector
models learn embeddings directly from raw text data.
  For word-level representations we propose Disambiguated Skip-gram: a neural
network model for learning multi-sense word embeddings. Representations learned
by this model can be used in downstream tasks, like part-of-speech tagging or
identification of semantic relations. In the word sense induction task
Disambiguated Skip-gram outperforms state-of-the-art models on three out of
four benchmarks datasets. Our model has an elegant probabilistic
interpretation. Furthermore, unlike previous models of this kind, it is
differentiable with respect to all its parameters and can be trained with
backpropagation. In addition to quantitative results, we present qualitative
evaluation of Disambiguated Skip-gram, including two-dimensional visualisations
of selected word-sense embeddings.

negative:
Question Answering with NLP has progressed through the evolution of advanced
model architectures like BERT and BiDAF and earlier word, character, and
context-based embeddings. As BERT has leapfrogged the accuracy of models, an
element of the next frontier can be the introduction of deep networks and an
effective way to train them. In this context, I explored the effectiveness of
deep networks focussing on the model encoder layer of BiDAF. BiDAF with its
heterogeneous layers provides the opportunity not only to explore the
effectiveness of deep networks but also to evaluate whether the refinements
made in lower layers are additive to the refinements made in the upper layers
of the model architecture. I believe the next greatest model in NLP will in
fact fold in a solid language modeling like BERT with a composite architecture
which will bring in refinements in addition to generic language modeling and
will have a more extensive layered architecture. I experimented with the Bypass
network, Residual Highway network, and DenseNet architectures. In addition, I
evaluated the effectiveness of ensembling the last few layers of the network. I
also studied the difference character embeddings make in adding them to the
word embeddings, and whether the effects are additive with deep networks. My
studies indicate that deep networks are in fact effective in giving a boost.
Also, the refinements in the lower layers like embeddings are passed on
additively to the gains made through deep networks.

negative:
We propose the Neural Vector Space Model (NVSM), a method that learns
representations of documents in an unsupervised manner for news article
retrieval. In the NVSM paradigm, we learn low-dimensional representations of
words and documents from scratch using gradient descent and rank documents
according to their similarity with query representations that are composed from
word representations. We show that NVSM performs better at document ranking
than existing latent semantic vector space methods. The addition of NVSM to a
mixture of lexical language models and a state-of-the-art baseline vector space
model yields a statistically significant increase in retrieval effectiveness.
Consequently, NVSM adds a complementary relevance signal. Next to semantic
matching, we find that NVSM performs well in cases where lexical matching is
needed.
  NVSM learns a notion of term specificity directly from the document
collection without feature engineering. We also show that NVSM learns
regularities related to Luhn significance. Finally, we give advice on how to
deploy NVSM in situations where model selection (e.g., cross-validation) is
infeasible. We find that an unsupervised ensemble of multiple models trained
with different hyperparameter values performs better than a single
cross-validated model. Therefore, NVSM can safely be used for ranking documents
without supervised relevance judgments.

negative:
Recent advances in language modeling using deep neural networks have shown
that these models learn representations, that vary with the network depth from
morphology to semantic relationships like co-reference. We apply pre-trained
language models to low-resource named entity recognition for Historic German.
We show on a series of experiments that character-based pre-trained language
models do not run into trouble when faced with low-resource datasets. Our
pre-trained character-based language models improve upon classical CRF-based
methods and previous work on Bi-LSTMs by boosting F1 score performance by up to
6%. Our pre-trained language and NER models are publicly available under
https://github.com/stefan-it/historic-ner .

negative:
Syntax has been demonstrated highly effective in neural machine translation
(NMT). Previous NMT models integrate syntax by representing 1-best tree outputs
from a well-trained parsing system, e.g., the representative Tree-RNN and
Tree-Linearization methods, which may suffer from error propagation. In this
work, we propose a novel method to integrate source-side syntax implicitly for
NMT. The basic idea is to use the intermediate hidden representations of a
well-trained end-to-end dependency parser, which are referred to as
syntax-aware word representations (SAWRs). Then, we simply concatenate such
SAWRs with ordinary word embeddings to enhance basic NMT models. The method can
be straightforwardly integrated into the widely-used sequence-to-sequence
(Seq2Seq) NMT models. We start with a representative RNN-based Seq2Seq baseline
system, and test the effectiveness of our proposed method on two benchmark
datasets of the Chinese-English and English-Vietnamese translation tasks,
respectively. Experimental results show that the proposed approach is able to
bring significant BLEU score improvements on the two datasets compared with the
baseline, 1.74 points for Chinese-English translation and 0.80 point for
English-Vietnamese translation, respectively. In addition, the approach also
outperforms the explicit Tree-RNN and Tree-Linearization methods.

negative:
Semantic parsing is the process of mapping a natural language sentence into a
formal representation of its meaning. In this work we use the neural network
approach to transform natural language sentence into a query to an ontology
database in the SPARQL language. This method does not rely on handcraft-rules,
high-quality lexicons, manually-built templates or other handmade complex
structures. Our approach is based on vector space model and neural networks.
The proposed model is based in two learning steps. The first step generates a
vector representation for the sentence in natural language and SPARQL query.
The second step uses this vector representation as input to a neural network
(LSTM with attention mechanism) to generate a model able to encode natural
language and decode SPARQL.

negative:
Recursive neural network models and their accompanying vector representations
for words have seen success in an array of increasingly semantically
sophisticated tasks, but almost nothing is known about their ability to
accurately capture the aspects of linguistic meaning that are necessary for
interpretation or reasoning. To evaluate this, I train a recursive model on a
new corpus of constructed examples of logical reasoning in short sentences,
like the inference of "some animal walks" from "some dog walks" or "some cat
walks," given that dogs and cats are animals. This model learns representations
that generalize well to new types of reasoning pattern in all but a few cases,
a result which is promising for the ability of learned representation models to
capture logical reasoning.

negative:
Semantic sentence embedding models encode natural language sentences into
vectors, such that closeness in embedding space indicates closeness in the
semantics between the sentences. Bilingual data offers a useful signal for
learning such embeddings: properties shared by both sentences in a translation
pair are likely semantic, while divergent properties are likely stylistic or
language-specific. We propose a deep latent variable model that attempts to
perform source separation on parallel sentences, isolating what they have in
common in a latent semantic vector, and explaining what is left over with
language-specific latent vectors. Our proposed approach differs from past work
on semantic sentence encoding in two ways. First, by using a variational
probabilistic framework, we introduce priors that encourage source separation,
and can use our model's posterior to predict sentence embeddings for
monolingual data at test time. Second, we use high-capacity transformers as
both data generating distributions and inference networks -- contrasting with
most past work on sentence embeddings. In experiments, our approach
substantially outperforms the state-of-the-art on a standard suite of
unsupervised semantic similarity evaluations. Further, we demonstrate that our
approach yields the largest gains on more difficult subsets of these
evaluations where simple word overlap is not a good indicator of similarity.

negative:
Much information available to applied researchers is contained within written
language or spoken text. Deep language models such as BERT have achieved
unprecedented success in many applications of computational linguistics.
However, much less is known about how these models can be used to analyze
existing text. We propose a novel method that combines transformer models with
network analysis to form a self-referential representation of language use
within a corpus of interest. Our approach produces linguistic relations
strongly consistent with the underlying model as well as mathematically
well-defined operations on them, while reducing the amount of discretionary
choices of representation and distance measures. It represents, to the best of
our knowledge, the first unsupervised method to extract semantic networks
directly from deep language models. We illustrate our approach in a semantic
analysis of the term "founder". Using the entire corpus of Harvard Business
Review from 1980 to 2020, we find that ties in our network track the semantics
of discourse over time, and across contexts, identifying and relating clusters
of semantic and syntactic relations. Finally, we discuss how this method can
also complement and inform analyses of the behavior of deep learning models.

negative:
Semantic representations have long been argued as potentially useful for
enforcing meaning preservation and improving generalization performance of
machine translation methods. In this work, we are the first to incorporate
information about predicate-argument structure of source sentences (namely,
semantic-role representations) into neural machine translation. We use Graph
Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders
and achieve improvements in BLEU scores over the linguistic-agnostic and
syntax-aware versions on the English--German language pair.

negative:
Humans can learn structural properties about a word from minimal experience,
and deploy their learned syntactic representations uniformly in different
grammatical contexts. We assess the ability of modern neural language models to
reproduce this behavior in English and evaluate the effect of structural
supervision on learning outcomes. First, we assess few-shot learning
capabilities by developing controlled experiments that probe models' syntactic
nominal number and verbal argument structure generalizations for tokens seen as
few as two times during training. Second, we assess invariance properties of
learned representation: the ability of a model to transfer syntactic
generalizations from a base context (e.g., a simple declarative active-voice
sentence) to a transformed context (e.g., an interrogative sentence). We test
four models trained on the same dataset: an n-gram baseline, an LSTM, and two
LSTM-variants trained with explicit structural supervision (Dyer et al.,2016;
Charniak et al., 2016). We find that in most cases, the neural models are able
to induce the proper syntactic generalizations after minimal exposure, often
from just two examples during training, and that the two structurally
supervised models generalize more accurately than the LSTM model. All neural
models are able to leverage information learned in base contexts to drive
expectations in transformed contexts, indicating that they have learned some
invariance properties of syntax.

negative:
We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual
information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM)
build vector-based word representations by learning to predict linguistic
contexts in text corpora. However, for a restricted set of words, the models
are also exposed to visual representations of the objects they denote
(extracted from natural images), and must predict linguistic and visual
features jointly. The MMSKIP-GRAM models achieve good performance on a variety
of semantic benchmarks. Moreover, since they propagate visual information to
all words, we use them to improve image labeling and retrieval in the zero-shot
setup, where the test concepts are never seen during model training. Finally,
the MMSKIP-GRAM models discover intriguing visual properties of abstract words,
paving the way to realistic implementations of embodied theories of meaning.

negative:
This paper introduces a sentence to vector encoding framework suitable for
advanced natural language processing. Our latent representation is shown to
encode sentences with common semantic information with similar vector
representations. The vector representation is extracted from an encoder-decoder
model which is trained on sentence paraphrase pairs. We demonstrate the
application of the sentence representations for two different tasks -- sentence
paraphrasing and paragraph summarization, making it attractive for commonly
used recurrent frameworks that process text. Experimental results help gain
insight how vector representations are suitable for advanced language
embedding.

negative:
The goal of semantic role labelling (SRL) is to recognise the
predicate-argument structure of a sentence. Recent models have shown that
syntactic information can enhance the SRL performance, but other
syntax-agnostic approaches achieved reasonable performance. The best way to
encode syntactic information for the SRL task is still an open question. In
this paper, we propose the Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr)
architecture, which encodes the syntactic structure with a novel way to input
graph relations as embeddings directly into the self-attention mechanism of
Transformer. This approach adds a soft bias towards attention patterns that
follow the syntactic structure but also allows the model to use this
information to learn alternative patterns. We evaluate our model on both
dependency-based and span-based SRL datasets, and outperform all previous
syntax-aware and syntax-agnostic models in both in-domain and out-of-domain
settings, on the CoNLL 2005 and CoNLL 2009 datasets. Our architecture is
general and can be applied to encode any graph information for a desired
downstream task.

negative:
Inspired by recent advances in multimodal learning and machine translation,
we introduce an encoder-decoder pipeline that learns (a): a multimodal joint
embedding space with images and text and (b): a novel language model for
decoding distributed representations from our space. Our pipeline effectively
unifies joint image-text embedding models with multimodal neural language
models. We introduce the structure-content neural language model that
disentangles the structure of a sentence to its content, conditioned on
representations produced by the encoder. The encoder allows one to rank images
and sentences while the decoder can generate novel descriptions from scratch.
Using LSTM to encode sentences, we match the state-of-the-art performance on
Flickr8K and Flickr30K without using object detections. We also set new best
results when using the 19-layer Oxford convolutional network. Furthermore we
show that with linear encoders, the learned embedding space captures multimodal
regularities in terms of vector space arithmetic e.g. *image of a blue car* -
"blue" + "red" is near images of red cars. Sample captions generated for 800
images are made available for comparison.

negative:
A conversational agent (chatbot) is a piece of software that is able to
communicate with humans using natural language. Modeling conversation is an
important task in natural language processing and artificial intelligence.
While chatbots can be used for various tasks, in general they have to
understand users' utterances and provide responses that are relevant to the
problem at hand.
  In my work, I conduct an in-depth survey of recent literature, examining over
70 publications related to chatbots published in the last 3 years. Then, I
proceed to make the argument that the very nature of the general conversation
domain demands approaches that are different from current state-of-of-the-art
architectures. Based on several examples from the literature I show why current
chatbot models fail to take into account enough priors when generating
responses and how this affects the quality of the conversation. In the case of
chatbots, these priors can be outside sources of information that the
conversation is conditioned on like the persona or mood of the conversers. In
addition to presenting the reasons behind this problem, I propose several ideas
on how it could be remedied.
  The next section focuses on adapting the very recent Transformer model to the
chatbot domain, which is currently state-of-the-art in neural machine
translation. I first present experiments with the vanilla model, using
conversations extracted from the Cornell Movie-Dialog Corpus. Secondly, I
augment the model with some of my ideas regarding the issues of encoder-decoder
architectures. More specifically, I feed additional features into the model
like mood or persona together with the raw conversation data. Finally, I
conduct a detailed analysis of how the vanilla model performs on conversational
data by comparing it to previous chatbot models and how the additional features
affect the quality of the generated responses.

negative:
In the last half-decade, the field of natural language processing (NLP) has
undergone two major transitions: the switch to neural networks as the primary
modeling paradigm and the homogenization of the training regime (pre-train,
then fine-tune). Amidst this process, language models have emerged as NLP's
workhorse, displaying increasingly fluent generation capabilities and proving
to be an indispensable means of knowledge transfer downstream. Due to the
otherwise opaque, black-box nature of such models, researchers have employed
aspects of linguistic theory in order to characterize their behavior. Questions
central to syntax -- the study of the hierarchical structure of language --
have factored heavily into such work, shedding invaluable insights about
models' inherent biases and their ability to make human-like generalizations.
In this paper, we attempt to take stock of this growing body of literature. In
doing so, we observe a lack of clarity across numerous dimensions, which
influences the hypotheses that researchers form, as well as the conclusions
they draw from their findings. To remedy this, we urge researchers make careful
considerations when investigating coding properties, selecting representations,
and evaluating via downstream tasks. Furthermore, we outline the implications
of the different types of research questions exhibited in studies on syntax, as
well as the inherent pitfalls of aggregate metrics. Ultimately, we hope that
our discussion adds nuance to the prospect of studying language models and
paves the way for a less monolithic perspective on syntax in this context.

negative:
Incorporating syntax into neural approaches in NLP has a multitude of
practical and scientific benefits. For instance, a language model that is
syntax-aware is likely to be able to produce better samples; even a
discriminative model like BERT with a syntax module could be used for core NLP
tasks like unsupervised syntactic parsing. Rapid progress in recent years was
arguably spurred on by the empirical success of the Parsing-Reading-Predict
architecture of (Shen et al., 2018a), later simplified by the Order Neuron LSTM
of (Shen et al., 2019). Most notably, this is the first time neural approaches
were able to successfully perform unsupervised syntactic parsing (evaluated by
various metrics like F-1 score).
  However, even heuristic (much less fully mathematical) understanding of why
and when these architectures work is lagging severely behind. In this work, we
answer representational questions raised by the architectures in (Shen et al.,
2018a, 2019), as well as some transition-based syntax-aware language models
(Dyer et al., 2016): what kind of syntactic structure can current neural
approaches to syntax represent? Concretely, we ground this question in the
sandbox of probabilistic context-free-grammars (PCFGs), and identify a key
aspect of the representational power of these approaches: the amount and
directionality of context that the predictor has access to when forced to make
parsing decision. We show that with limited context (either bounded, or
unidirectional), there are PCFGs, for which these approaches cannot represent
the max-likelihood parse; conversely, if the context is unlimited, they can
represent the max-likelihood parse of any PCFG.

negative:
The field of natural language understanding has experienced exponential
progress in the last few years, with impressive results in several tasks. This
success has motivated researchers to study the underlying knowledge encoded by
these models. Despite this, attempts to understand their semantic capabilities
have not been successful, often leading to non-conclusive, or contradictory
conclusions among different works. Via a probing classifier, we extract the
underlying knowledge graph of nine of the most influential language models of
the last years, including word embeddings, text generators, and context
encoders. This probe is based on concept relatedness, grounded on WordNet. Our
results reveal that all the models encode this knowledge, but suffer from
several inaccuracies. Furthermore, we show that the different architectures and
training strategies lead to different model biases. We conduct a systematic
evaluation to discover specific factors that explain why some concepts are
challenging. We hope our insights will motivate the development of models that
capture concepts more precisely.

negative:
At present, different deep learning models are presenting high accuracy on
popular inference datasets such as SNLI, MNLI, and SciTail. However, there are
different indicators that those datasets can be exploited by using some simple
linguistic patterns. This fact poses difficulties to our understanding of the
actual capacity of machine learning models to solve the complex task of textual
inference. We propose a new set of syntactic tasks focused on contradiction
detection that require specific capacities over linguistic logical forms such
as: Boolean coordination, quantifiers, definite description, and counting
operators. We evaluate two kinds of deep learning models that implicitly
exploit language structure: recurrent models and the Transformer network BERT.
We show that although BERT is clearly more efficient to generalize over most
logical forms, there is space for improvement when dealing with counting
operators. Since the syntactic tasks can be implemented in different languages,
we show a successful case of cross-lingual transfer learning between English
and Portuguese.

negative:
We propose transfer learning as a method for analyzing the encoding of
grammatical structure in neural language models. We train LSTMs on
non-linguistic data and evaluate their performance on natural language to
assess which kinds of data induce generalizable structural features that LSTMs
can use for natural language. We find that training on non-linguistic data with
latent structure (MIDI music or Java code) improves test performance on natural
language, despite no overlap in surface form or vocabulary. To pinpoint the
kinds of abstract structure that models may be encoding to lead to this
improvement, we run similar experiments with two artificial parentheses
languages: one which has a hierarchical recursive structure, and a control
which has paired tokens but no recursion. Surprisingly, training a model on
either of these artificial languages leads to the same substantial gains when
testing on natural language. Further experiments on transfer between natural
languages controlling for vocabulary overlap show that zero-shot performance on
a test language is highly correlated with typological syntactic similarity to
the training language, suggesting that representations induced by pre-training
correspond to the cross-linguistic syntactic properties. Our results provide
insights into the ways that neural models represent abstract syntactic
structure, and also about the kind of structural inductive biases which allow
for natural language acquisition.

negative:
In this work we propose a simple and efficient framework for learning
sentence representations from unlabelled data. Drawing inspiration from the
distributional hypothesis and recent work on learning sentence representations,
we reformulate the problem of predicting the context in which a sentence
appears as a classification problem. Given a sentence and its context, a
classifier distinguishes context sentences from other contrastive sentences
based on their vector representations. This allows us to efficiently learn
different types of encoding functions, and we show that the model learns
high-quality sentence representations. We demonstrate that our sentence
representations outperform state-of-the-art unsupervised and supervised
representation learning methods on several downstream NLP tasks that involve
understanding sentence semantics while achieving an order of magnitude speedup
in training time.

negative:
Long-range semantic coherence remains a challenge in automatic language
generation and understanding. We demonstrate that large language models have
insufficiently learned the effect of distant words on next-token prediction. We
present coherence boosting, an inference procedure that increases a LM's focus
on a long context. We show the benefits of coherence boosting with pretrained
models by distributional analyses of generated ordinary text and dialog
responses. It is also found that coherence boosting with state-of-the-art
models for various zero-shot NLP tasks yields performance gains with no
additional training.

negative:
In this paper we investigate the linguistic knowledge learned by a Neural
Language Model (NLM) before and after a fine-tuning process and how this
knowledge affects its predictions during several classification problems. We
use a wide set of probing tasks, each of which corresponds to a distinct
sentence-level feature extracted from different levels of linguistic
annotation. We show that BERT is able to encode a wide range of linguistic
characteristics, but it tends to lose this information when trained on specific
downstream tasks. We also find that BERT's capacity to encode different kind of
linguistic properties has a positive influence on its predictions: the more it
stores readable linguistic information of a sentence, the higher will be its
capacity of predicting the expected label assigned to that sentence.

negative:
Measuring text complexity is an essential task in several fields and
applications (such as NLP, semantic web, smart education, etc.). The semantic
layer of text is more tacit than its syntactic structure and, as a result,
calculation of semantic complexity is more difficult than syntactic complexity.
While there are famous and powerful academic and commercial syntactic
complexity measures, the problem of measuring semantic complexity is still a
challenging one. In this paper, we introduce the DAST model, which stands for
Deciding About Semantic Complexity of a Text. DAST proposes an intuitionistic
approach to semantics that lets us have a well-defined model for the semantics
of a text and its complexity: semantic is considered as a lattice of intuitions
and, as a result, semantic complexity is defined as the result of a calculation
on this lattice. A set theoretic formal definition of semantic complexity, as a
6-tuple formal system, is provided. By using this formal system, a method for
measuring semantic complexity is presented. The evaluation of the proposed
approach is done by a set of three human-judgment experiments. The results show
that DAST model is capable of deciding about semantic complexity of text.
Furthermore, the analysis of the results leads us to introduce a Markovian
model for the process of common-sense, multiple-steps and semantic-complexity
reasoning in people. The results of Experiments demonstrate that our method
outperforms the random baseline with improvement in better precision and
competes with other methods by less error percentage.

negative:
Statistical methods have been widely employed in recent years to grasp many
language properties. The application of such techniques have allowed an
improvement of several linguistic applications, which encompasses machine
translation, automatic summarization and document classification. In the
latter, many approaches have emphasized the semantical content of texts, as it
is the case of bag-of-word language models. This approach has certainly yielded
reasonable performance. However, some potential features such as the structural
organization of texts have been used only on a few studies. In this context, we
probe how features derived from textual structure analysis can be effectively
employed in a classification task. More specifically, we performed a supervised
classification aiming at discriminating informative from imaginative documents.
Using a networked model that describes the local topological/dynamical
properties of function words, we achieved an accuracy rate of up to 95%, which
is much higher than similar networked approaches. A systematic analysis of
feature relevance revealed that symmetry and accessibility measurements are
among the most prominent network measurements. Our results suggest that these
measurements could be used in related language applications, as they play a
complementary role in characterizing texts.

negative:
A desideratum of high-quality translation systems is that they preserve
meaning, in the sense that two sentences with different meanings should not
translate to one and the same sentence in another language. However,
state-of-the-art systems often fail in this regard, particularly in cases where
the source and target languages partition the "meaning space" in different
ways. For instance, "I cut my finger." and "I cut my finger off." describe
different states of the world but are translated to French (by both Fairseq and
Google Translate) as "Je me suis coupe le doigt.", which is ambiguous as to
whether the finger is detached. More generally, translation systems are
typically many-to-one (non-injective) functions from source to target language,
which in many cases results in important distinctions in meaning being lost in
translation. Building on Bayesian models of informative utterance production,
we present a method to define a less ambiguous translation system in terms of
an underlying pre-trained neural sequence-to-sequence model. This method
increases injectivity, resulting in greater preservation of meaning as measured
by improvement in cycle-consistency, without impeding translation quality
(measured by BLEU score).

negative:
Pre-trained language models have achieved huge success on a wide range of NLP
tasks. However, contextual representations from pre-trained models contain
entangled semantic and syntactic information, and therefore cannot be directly
used to derive useful semantic sentence embeddings for some tasks. Paraphrase
pairs offer an effective way of learning the distinction between semantics and
syntax, as they naturally share semantics and often vary in syntax. In this
work, we present ParaBART, a semantic sentence embedding model that learns to
disentangle semantics and syntax in sentence embeddings obtained by pre-trained
language models. ParaBART is trained to perform syntax-guided paraphrasing,
based on a source sentence that shares semantics with the target paraphrase,
and a parse tree that specifies the target syntax. In this way, ParaBART learns
disentangled semantic and syntactic representations from their respective
inputs with separate encoders. Experiments in English show that ParaBART
outperforms state-of-the-art sentence embedding models on unsupervised semantic
similarity tasks. Additionally, we show that our approach can effectively
remove syntactic information from semantic sentence embeddings, leading to
better robustness against syntactic variation on downstream semantic tasks.

negative:
We propose two methods of learning vector representations of words and
phrases that each combine sentence context with structural features extracted
from dependency trees. Using several variations of neural network classifier,
we show that these combined methods lead to improved performance when used as
input features for supervised term-matching.

negative:
Neural network-based generative language models like ELMo and BERT can work
effectively as general purpose sentence encoders in text classification without
further fine-tuning. Is it possible to adapt them in a similar way for use as
general-purpose decoders? For this to be possible, it would need to be the case
that for any target sentence of interest, there is some continuous
representation that can be passed to the language model to cause it to
reproduce that sentence. We set aside the difficult problem of designing an
encoder that can produce such representations and, instead, ask directly
whether such representations exist at all. To do this, we introduce a pair of
effective, complementary methods for feeding representations into pretrained
unconditional language models and a corresponding set of methods to map
sentences into and out of this representation space, the reparametrized
sentence space. We then investigate the conditions under which a language model
can be made to generate a sentence through the identification of a point in
such a space and find that it is possible to recover arbitrary sentences nearly
perfectly with language models and representations of moderate size without
modifying any model parameters.

negative:
One central mystery of neural NLP is what neural models "know" about their
subject matter. When a neural machine translation system learns to translate
from one language to another, does it learn the syntax or semantics of the
languages? Can this knowledge be extracted from the system to fill holes in
human scientific knowledge? Existing typological databases contain relatively
full feature specifications for only a few hundred languages. Exploiting the
existence of parallel texts in more than a thousand languages, we build a
massive many-to-one neural machine translation (NMT) system from 1017 languages
into English, and use this to predict information missing from typological
databases. Experiments show that the proposed method is able to infer not only
syntactic, but also phonological and phonetic inventory features, and improves
over a baseline that has access to information about the languages' geographic
and phylogenetic neighbors.

negative:
Understanding the semantic relationships between terms is a fundamental task
in natural language processing applications. While structured resources that
can express those relationships in a formal way, such as ontologies, are still
scarce, a large number of linguistic resources gathering dictionary definitions
is becoming available, but understanding the semantic structure of natural
language definitions is fundamental to make them useful in semantic
interpretation tasks. Based on an analysis of a subset of WordNet's glosses, we
propose a set of semantic roles that compose the semantic structure of a
dictionary definition, and show how they are related to the definition's
syntactic configuration, identifying patterns that can be used in the
development of information extraction frameworks and semantic models.

negative:
One of the most complex syntactic representations used in computational
linguistics and NLP are discontinuous constituent trees, crucial for
representing all grammatical phenomena of languages such as German. Recent
advances in dependency parsing have shown that Pointer Networks excel in
efficiently parsing syntactic relations between words in a sentence. This kind
of sequence-to-sequence models achieve outstanding accuracies in building
non-projective dependency trees, but its potential has not been proved yet on a
more difficult task. We propose a novel neural network architecture that, by
means of Pointer Networks, is able to generate the most accurate discontinuous
constituent representations to date, even without the need of Part-of-Speech
tagging information. To do so, we internally model discontinuous constituent
structures as augmented non-projective dependency structures. The proposed
approach achieves state-of-the-art results on the two widely-used NEGRA and
TIGER benchmarks, outperforming previous work by a wide margin.

negative:
Current approaches to learning vector representations of text that are
compatible between different languages usually require some amount of parallel
text, aligned at word, sentence or at least document level. We hypothesize
however, that different natural languages share enough semantic structure that
it should be possible, in principle, to learn compatible vector representations
just by analyzing the monolingual distribution of words.
  In order to evaluate this hypothesis, we propose a scheme to map word vectors
trained on a source language to vectors semantically compatible with word
vectors trained on a target language using an adversarial autoencoder.
  We present preliminary qualitative results and discuss possible future
developments of this technique, such as applications to cross-lingual sentence
representations.

negative:
There is growing evidence that the classical notion of adversarial robustness
originally introduced for images has been adopted as a de facto standard by a
large part of the NLP research community. We show that this notion is
problematic in the context of NLP as it considers a narrow spectrum of
linguistic phenomena. In this paper, we argue for semantic robustness, which is
better aligned with the human concept of linguistic fidelity. We characterize
semantic robustness in terms of biases that it is expected to induce in a
model. We study semantic robustness of a range of vanilla and robustly trained
architectures using a template-based generative test bed. We complement the
analysis with empirical evidence that, despite being harder to implement,
semantic robustness can improve performance %gives guarantees for on complex
linguistic phenomena where models robust in the classical sense fail.

negative:
We investigated the impact of noisy linguistic features on the performance of
a Japanese speech synthesis system based on neural network that uses WaveNet
vocoder. We compared an ideal system that uses manually corrected linguistic
features including phoneme and prosodic information in training and test sets
against a few other systems that use corrupted linguistic features. Both
subjective and objective results demonstrate that corrupted linguistic
features, especially those in the test set, affected the ideal system's
performance significantly in a statistical sense due to a mismatched condition
between the training and test sets. Interestingly, while an utterance-level
Turing test showed that listeners had a difficult time differentiating
synthetic speech from natural speech, it further indicated that adding noise to
the linguistic features in the training set can partially reduce the effect of
the mismatch, regularize the model, and help the system perform better when
linguistic features of the test set are noisy.

negative:
The task of linearization is to find a grammatical order given a set of
words. Traditional models use statistical methods. Syntactic linearization
systems, which generate a sentence along with its syntactic tree, have shown
state-of-the-art performance. Recent work shows that a multi-layer LSTM
language model outperforms competitive statistical syntactic linearization
systems without using syntax. In this paper, we study neural syntactic
linearization, building a transition-based syntactic linearizer leveraging a
feed-forward neural network, observing significantly better results compared to
LSTM language models on this task.

negative:
Labeling explicit discourse relations is one of the most challenging
sub-tasks of the shallow discourse parsing where the goal is to identify the
discourse connectives and the boundaries of their arguments. The
state-of-the-art models achieve slightly above 45% of F-score by using
hand-crafted features. The current paper investigates the efficacy of the
pre-trained language models in this task. We find that the pre-trained language
models, when finetuned, are powerful enough to replace the linguistic features.
We evaluate our model on PDTB 2.0 and report the state-of-the-art results in
the extraction of the full relation. This is the first time when a model
outperforms the knowledge intensive models without employing any linguistic
features.

negative:
We introduce the syntactic scaffold, an approach to incorporating syntactic
information into semantic tasks. Syntactic scaffolds avoid expensive syntactic
processing at runtime, only making use of a treebank during training, through a
multitask objective. We improve over strong baselines on PropBank semantics,
frame semantics, and coreference resolution, achieving competitive performance
on all three tasks.

negative:
Semantic parsing, i.e., the automatic derivation of meaning representation
such as an instantiated predicate-argument structure for a sentence, plays a
critical role in deep processing of natural language. Unlike all other top
systems of semantic dependency parsing that have to rely on a pipeline
framework to chain up a series of submodels each specialized for a specific
subtask, the one presented in this article integrates everything into one
model, in hopes of achieving desirable integrity and practicality for real
applications while maintaining a competitive performance. This integrative
approach tackles semantic parsing as a word pair classification problem using a
maximum entropy classifier. We leverage adaptive pruning of argument candidates
and large-scale feature selection engineering to allow the largest feature
space ever in use so far in this field, it achieves a state-of-the-art
performance on the evaluation data set for CoNLL-2008 shared task, on top of
all but one top pipeline system, confirming its feasibility and effectiveness.

negative:
We introduce semantic form mid-tuning, an approach for transferring semantic
knowledge from semantic meaning representations into transformer-based language
encoders. In mid-tuning, we learn to align the text of general sentences -- not
tied to any particular inference task -- and structured semantic
representations of those sentences. Our approach does not require gold
annotated semantic representations. Instead, it makes use of automatically
generated semantic representations, such as from off-the-shelf PropBank and
FrameNet semantic parsers. We show that this alignment can be learned
implicitly via classification or directly via triplet loss. Our method yields
language encoders that demonstrate improved predictive performance across
inference, reading comprehension, textual similarity, and other semantic tasks
drawn from the GLUE, SuperGLUE, and SentEval benchmarks. We evaluate our
approach on three popular baseline models, where our experimental results and
analysis concludes that current pre-trained language models can further benefit
from structured semantic frames with the proposed mid-tuning method, as they
inject additional task-agnostic knowledge to the encoder, improving the
generated embeddings as well as the linguistic properties of the given model,
as evident from improvements on a popular sentence embedding toolkit and a
variety of probing tasks.

negative:
As a fundamental NLP task, semantic role labeling (SRL) aims to discover the
semantic roles for each predicate within one sentence. This paper investigates
how to incorporate syntactic knowledge into the SRL task effectively. We
present different approaches of encoding the syntactic information derived from
dependency trees of different quality and representations; we propose a
syntax-enhanced self-attention model and compare it with other two strong
baseline methods; and we conduct experiments with newly published deep
contextualized word representations as well. The experiment results demonstrate
that with proper incorporation of the high quality syntactic information, our
model achieves a new state-of-the-art performance for the Chinese SRL task on
the CoNLL-2009 dataset.

negative:
Answer sentence selection is the task of identifying sentences that contain
the answer to a given question. This is an important problem in its own right
as well as in the larger context of open domain question answering. We propose
a novel approach to solving this task via means of distributed representations,
and learn to match questions with answers by considering their semantic
encoding. This contrasts prior work on this task, which typically relies on
classifiers with large numbers of hand-crafted syntactic and semantic features
and various external resources. Our approach does not require any feature
engineering nor does it involve specialist linguistic data, making this model
easily applicable to a wide range of domains and languages. Experimental
results on a standard benchmark dataset from TREC demonstrate that---despite
its simplicity---our model matches state of the art performance on the answer
sentence selection task.

negative:
In many multilingual text classification problems, the documents in different
languages often share the same set of categories. To reduce the labeling cost
of training a classification model for each individual language, it is
important to transfer the label knowledge gained from one language to another
language by conducting cross language classification. In this paper we develop
a novel subspace co-regularized multi-view learning method for cross language
text classification. This method is built on parallel corpora produced by
machine translation. It jointly minimizes the training error of each classifier
in each language while penalizing the distance between the subspace
representations of parallel documents. Our empirical study on a large set of
cross language text classification tasks shows the proposed method consistently
outperforms a number of inductive methods, domain adaptation methods, and
multi-view learning methods.

negative:
This paper investigates the ability of artificial neural networks to judge
the grammatical acceptability of a sentence, with the goal of testing their
linguistic competence. We introduce the Corpus of Linguistic Acceptability
(CoLA), a set of 10,657 English sentences labeled as grammatical or
ungrammatical from published linguistics literature. As baselines, we train
several recurrent neural network models on acceptability classification, and
find that our models outperform unsupervised models by Lau et al (2016) on
CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et
al.'s models and ours learn systematic generalizations like subject-verb-object
order. However, all models we test perform far below human level on a wide
range of grammatical constructions.

negative:
Semantic role labeling (SRL) is to recognize the predicate-argument structure
of a sentence, including subtasks of predicate disambiguation and argument
labeling. Previous studies usually formulate the entire SRL problem into two or
more subtasks. For the first time, this paper introduces an end-to-end neural
model which unifiedly tackles the predicate disambiguation and the argument
labeling in one shot. Using a biaffine scorer, our model directly predicts all
semantic role labels for all given word pairs in the sentence without relying
on any syntactic parse information. Specifically, we augment the BiLSTM encoder
with a non-linear transformation to further distinguish the predicate and the
argument in a given sentence, and model the semantic role labeling process as a
word pair classification task by employing the biaffine attentional mechanism.
Though the proposed model is syntax-agnostic with local decoder, it outperforms
the state-of-the-art syntax-aware SRL systems on the CoNLL-2008, 2009
benchmarks for both English and Chinese. To our best knowledge, we report the
first syntax-agnostic SRL model that surpasses all known syntax-aware models.

negative:
Neural language models learn, to varying degrees of accuracy, the grammatical
properties of natural languages. In this work, we investigate whether there are
systematic sources of variation in the language models' accuracy. Focusing on
subject-verb agreement and reflexive anaphora, we find that certain nouns are
systematically understood better than others, an effect which is robust across
grammatical tasks and different language models. Surprisingly, we find that
across four orders of magnitude, corpus frequency is unrelated to a noun's
performance on grammatical tasks. Finally, we find that a novel noun's
grammatical properties can be few-shot learned from various types of training
data. The results present a paradox: there should be less variation in
grammatical performance than is actually observed.

negative:
Document categorization, which aims to assign a topic label to each document,
plays a fundamental role in a wide variety of applications. Despite the success
of existing studies in conventional supervised document classification, they
are less concerned with two real problems: (1) the presence of metadata: in
many domains, text is accompanied by various additional information such as
authors and tags. Such metadata serve as compelling topic indicators and should
be leveraged into the categorization framework; (2) label scarcity: labeled
training samples are expensive to obtain in some cases, where categorization
needs to be performed using only a small set of annotated data. In recognition
of these two challenges, we propose MetaCat, a minimally supervised framework
to categorize text with metadata. Specifically, we develop a generative process
describing the relationships between words, documents, labels, and metadata.
Guided by the generative model, we embed text and metadata into the same
semantic space to encode heterogeneous signals. Then, based on the same
generative process, we synthesize training samples to address the bottleneck of
label scarcity. We conduct a thorough evaluation on a wide range of datasets.
Experimental results prove the effectiveness of MetaCat over many competitive
baselines.

negative:
Organizing complex peer production projects and advancing scientific
knowledge of open collaboration each depend on the ability to measure quality.
Article quality ratings on English language Wikipedia have been widely used by
both Wikipedia community members and academic researchers for purposes like
tracking knowledge gaps and studying how political polarization shapes
collaboration. Even so, measuring quality presents many methodological
challenges. The most widely used systems use labels on discrete ordinal scales
when assessing quality, but such labels can be inconvenient for statistics and
machine learning. Prior work handles this by assuming that different levels of
quality are "evenly spaced" from one another. This assumption runs counter to
intuitions about the relative degrees of effort needed to raise Wikipedia
encyclopedia articles to different quality levels. Furthermore, models from
prior work are fit to datasets that oversample high-quality articles. This
limits their accuracy for representative samples of articles or revisions. I
describe a technique extending the Wikimedia Foundations' ORES article quality
model to address these limitations. My method uses weighted ordinal regression
models to construct one-dimensional continuous measures of quality. While
scores from my technique and from prior approaches are correlated, my approach
improves accuracy for research datasets and provides evidence that the "evenly
spaced" assumption is unfounded in practice on English Wikipedia. I conclude
with recommendations for using quality scores in future research and include
the full code, data, and models.

negative:
My doctoral research focuses on understanding semantic knowledge in neural
network models trained solely to predict natural language (referred to as
language models, or LMs), by drawing on insights from the study of concepts and
categories grounded in cognitive science. I propose a framework inspired by
'inductive reasoning,' a phenomenon that sheds light on how humans utilize
background knowledge to make inductive leaps and generalize from new pieces of
information about concepts and their properties. Drawing from experiments that
study inductive reasoning, I propose to analyze semantic inductive
generalization in LMs using phenomena observed in human-induction literature,
investigate inductive behavior on tasks such as implicit reasoning and emergent
feature recognition, and analyze and relate induction dynamics to the learned
conceptual representation space.

negative:
Media coverage has a substantial effect on the public perception of events.
Nevertheless, media outlets are often biased. One way to bias news articles is
by altering the word choice. The automatic identification of bias by word
choice is challenging, primarily due to the lack of gold-standard data sets and
high context dependencies. In this research project, I aim to devise data sets
and methods to identify media bias. To achieve this, I plan to research methods
using natural language processing and deep learning while employing models and
using analysis concepts from psychology and linguistics. The first results
indicate the effectiveness of an interdisciplinary research approach. My vision
is to devise a system that helps news readers become aware of media coverage
differences caused by bias. So far, my best performing BERT-based model is
pre-trained on a larger corpus consisting of distant labels, indicating that
distant supervision has the potential to become a solution for the difficult
task of bias detection.

negative:
Vector representations have become a central element in semantic language
modelling, leading to mathematical overlaps with many fields including quantum
theory. Compositionality is a core goal for such representations: given
representations for 'wet' and 'fish', how should the concept 'wet fish' be
represented?
  This position paper surveys this question from two points of view. The first
considers the question of whether an explicit mathematical representation can
be successful using only tools from within linear algebra, or whether other
mathematical tools are needed. The second considers whether semantic vector
composition should be explicitly described mathematically, or whether it can be
a model-internal side-effect of training a neural network.
  A third and newer question is whether a compositional model can be
implemented on a quantum computer. Given the fundamentally linear nature of
quantum mechanics, we propose that these questions are related, and that this
survey may help to highlight candidate operations for future quantum
implementation.

negative:
We reduce the task of (span-based) PropBank-style semantic role labeling
(SRL) to syntactic dependency parsing. Our approach is motivated by our
empirical analysis that shows three common syntactic patterns account for over
98% of the SRL annotations for both English and Chinese data. Based on this
observation, we present a conversion scheme that packs SRL annotations into
dependency tree representations through joint labels that permit highly
accurate recovery back to the original format. This representation allows us to
train statistical dependency parsers to tackle SRL and achieve competitive
performance with the current state of the art. Our findings show the promise of
syntactic dependency trees in encoding semantic role relations within their
syntactic domain of locality, and point to potential further integration of
syntactic methods into semantic role labeling in the future.

negative:
Neural network based models are a very powerful tool for creating word
embeddings, the objective of these models is to group similar words together.
These embeddings have been used as features to improve results in various
applications such as document classification, named entity recognition, etc.
Neural language models are able to learn word representations which have been
used to capture semantic shifts across time and geography. The objective of
this paper is to first identify and then visualize how words change meaning in
different text corpus. We will train a neural language model on texts from a
diverse set of disciplines philosophy, religion, fiction etc. Each text will
alter the embeddings of the words to represent the meaning of the word inside
that text. We will present a computational technique to detect words that
exhibit significant linguistic shift in meaning and usage. We then use enhanced
scatterplots and storyline visualization to visualize the linguistic shift.

negative:
Predicting the structure of a discourse is challenging because relations
between discourse segments are often implicit and thus hard to distinguish
computationally. I extend previous work to classify implicit discourse
relations by introducing a novel set of features on the level of semantic
roles. My results demonstrate that such features are helpful, yielding results
competitive with other feature-rich approaches on the PDTB. My main
contribution is an analysis of improvements that can be traced back to
role-based features, providing insights into why and when role semantics is
helpful.

negative:
To investigate the evolution of syntax, we need to ascertain the evolutionary
r\^ole of syntax and, before that, the very nature of syntax. Here, we will
assume that syntax is computing. And then, since we are computationally Turing
complete, we meet an evolutionary anomaly, the anomaly of sytax: we are
syntactically too competent for syntax. Assuming that problem solving is
computing, and realizing that the evolutionary advantage of Turing completeness
is full problem solving and not syntactic proficiency, we explain the anomaly
of syntax by postulating that syntax and problem solving co-evolved in humans
towards Turing completeness. Examining the requirements that full problem
solving impose on language, we find firstly that semantics is not sufficient
and that syntax is necessary to represent problems. Our final conclusion is
that full problem solving requires a functional semantics on an infinite
tree-structured syntax. Besides these results, the introduction of Turing
completeness and problem solving to explain the evolution of syntax should help
us to fit the evolution of language within the evolution of cognition, giving
us some new clues to understand the elusive relation between language and
thinking.

negative:
The growing popularity of Virtual Assistants poses new challenges for Entity
Resolution, the task of linking mentions in text to their referent entities in
a knowledge base. Specifically, in the shopping domain, customers tend to use
implicit utterances (e.g., "organic milk") rather than explicit names, leading
to a large number of candidate products. Meanwhile, for the same query,
different customers may expect different results. For example, with "add milk
to my cart", a customer may refer to a certain organic product, while some
customers may want to re-order products they regularly purchase. To address
these issues, we propose a new framework that leverages personalized features
to improve the accuracy of product ranking. We first build a cross-source
heterogeneous knowledge graph from customer purchase history and product
knowledge graph to jointly learn customer and product embeddings. After that,
we incorporate product, customer, and history representations into a neural
reranking model to predict which candidate is most likely to be purchased for a
specific customer. Experiments show that our model substantially improves the
accuracy of the top ranked candidates by 24.6% compared to the state-of-the-art
product search model.

negative:
In Computer-Supported learning, monitoring and engaging a group of learners
is a complex task for teachers, especially when learners are working
collaboratively: Are my students motivated? What kind of progress are they
making? Should I intervene? Is my communication and the didactic design adapted
to my students? Our hypothesis is that the analysis of natural language
interactions between students, and between students and teachers, provide very
valuable information and could be used to produce qualitative indicators to
help teachers' decisions. We develop an automatic approach in three steps (1)
to explore the discursive functions of messages in a CSCL platform, (2) to
classify the messages automatically and (3) to evaluate correlations between
discursive attitudes and other variables linked to the learning activity.
Results tend to show that some types of discourse are correlated with a notion
of Progress on the learning activities and the importance of emotive
participation from the Teacher.

negative:
Semantic measures are widely used today to estimate the strength of the
semantic relationship between elements of various types: units of language
(e.g., words, sentences, documents), concepts or even instances semantically
characterized (e.g., diseases, genes, geographical locations). Semantic
measures play an important role to compare such elements according to semantic
proxies: texts and knowledge representations, which support their meaning or
describe their nature. Semantic measures are therefore essential for designing
intelligent agents which will for example take advantage of semantic analysis
to mimic human ability to compare abstract or concrete objects. This paper
proposes a comprehensive survey of the broad notion of semantic measure for the
comparison of units of language, concepts or instances based on semantic proxy
analyses. Semantic measures generalize the well-known notions of semantic
similarity, semantic relatedness and semantic distance, which have been
extensively studied by various communities over the last decades (e.g.,
Cognitive Sciences, Linguistics, and Artificial Intelligence to mention a few).

negative:
Recursive noun phrases (NPs) have interesting semantic properties. For
example, "my favorite new movie" is not necessarily my favorite movie, whereas
"my new favorite movie" is. This is common sense to humans, yet it is unknown
whether language models have such knowledge. We introduce the Recursive Noun
Phrase Challenge (RNPC), a dataset of three textual inference tasks involving
textual entailment and event plausibility comparison, precisely targeting the
understanding of recursive NPs. When evaluated on RNPC, state-of-the-art
Transformer models only perform around chance. Still, we show that such
knowledge is learnable with appropriate data. We further probe the models for
relevant linguistic features that can be learned from our tasks, including
modifier semantic category and modifier scope. Finally, models trained on RNPC
achieve strong zero-shot performance on an extrinsic Harm Detection evaluation
task, showing the usefulness of the understanding of recursive NPs in
downstream applications.

negative:
Emotion stimulus extraction is a fine-grained subtask of emotion analysis
that focuses on identifying the description of the cause behind an emotion
expression from a text passage (e.g., in the sentence "I am happy that I passed
my exam" the phrase "passed my exam" corresponds to the stimulus.). Previous
work mainly focused on Mandarin and English, with no resources or models for
German. We fill this research gap by developing a corpus of 2006 German news
headlines annotated with emotions and 811 instances with annotations of
stimulus phrases. Given that such corpus creation efforts are time-consuming
and expensive, we additionally work on an approach for projecting the existing
English GoodNewsEveryone (GNE) corpus to a machine-translated German version.
We compare the performance of a conditional random field (CRF) model (trained
monolingually on German and cross-lingually via projection) with a multilingual
XLM-RoBERTa (XLM-R) model. Our results show that training with the German
corpus achieves higher F1 scores than projection. Experiments with XLM-R
outperform their respective CRF counterparts.

negative:
The question of how to probe contextual word representations for linguistic
structure in a way that is both principled and useful has seen significant
attention recently in the NLP literature. In our contribution to this
discussion, we argue for a probe metric that reflects the fundamental trade-off
between probe complexity and performance: the Pareto hypervolume. To measure
complexity, we present a number of parametric and non-parametric metrics. Our
experiments using Pareto hypervolume as an evaluation metric show that probes
often do not conform to our expectations---e.g., why should the non-contextual
fastText representations encode more morpho-syntactic information than the
contextual BERT representations? These results suggest that common, simplistic
probing tasks, such as part-of-speech labeling and dependency arc labeling, are
inadequate to evaluate the linguistic structure encoded in contextual word
representations. This leads us to propose full dependency parsing as a probing
task. In support of our suggestion that harder probing tasks are necessary, our
experiments with dependency parsing reveal a wide gap in syntactic knowledge
between contextual and non-contextual representations.

negative:
We present a dataset for evaluating the grammaticality of the predictions of
a language model. We automatically construct a large number of minimally
different pairs of English sentences, each consisting of a grammatical and an
ungrammatical sentence. The sentence pairs represent different variations of
structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and
negative polarity items. We expect a language model to assign a higher
probability to the grammatical sentence than the ungrammatical one. In an
experiment using this data set, an LSTM language model performed poorly on many
of the constructions. Multi-task training with a syntactic objective (CCG
supertagging) improved the LSTM's accuracy, but a large gap remained between
its performance and the accuracy of human participants recruited online. This
suggests that there is considerable room for improvement over LSTMs in
capturing syntax in a language model.

negative:
Basic body-part names (BBPNs) were defined as body-part names in Swadesh
basic 200 words. Non-Mayan cognates of Mayan (MY) BBPNs were extensively
searched for, by comparing with non-MY vocabulary, including ca.1300 basic
words of 82 AN languages listed by Tryon (1985), etc. Thus found cognates (CGs)
in non-MY are listed in Table 1, as classified by language groups to which most
similar cognates (MSCs) of MY BBPNs belong. CGs of MY are classified to 23
mutually unrelated CG-items, of which 17.5 CG-items have their MSCs in
Austronesian (AN), giving its closest similarity score (CSS), CSS(AN) = 17.5,
which consists of 10.33 MSCs in Formosan, 1.83 MSCs in Western
Malayo-Polynesian (W.MP), 0.33 in Central MP, 0.0 in SHWNG, and 5.0 in Oceanic
[i.e., CSS(FORM)= 10.33, CSS(W.MP) = 1.88, ..., CSS(OC)= 5.0]. These CSSs for
language (sub)groups are also listed in the underline portion of every section
of (Section1 - Section 6) in Table 1. Chi-squar test (degree of freedom = 1)
using [Eq 1] and [Eqs.2] revealed that MSCs of MY BBPNs are distributed in
Formosan in significantly higher frequency (P < 0.001) than in other subgroups
of AN, as well as than in non-AN languages. MY is thus concluded to have been
derived from Formosan of AN. Eskimo shows some BBPN similarities to FORM and
MY.