query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I am seeking to develop a method that yields a range of promising output candidates rather than a single solution, a concept known as conformal prediction, in scenarios where data availability for the target task is limited. I need a model capable of automatically analyzing task features and identifying similar tasks with sufficiently available training data. This model should then discern the differences between tasks and transfer knowledge from similar tasks to new ones. It should also retain the candidate that the user ultimately selects, storing this data under the new task for future reference. I am currently undecided on whether to use a transformer or a convolutional neural network (CNN) for this model's implementation, so I would appreciate a comparison of these two structures. Additionally, I would like examples demonstrating fields where conformal prediction outperforms the conventional best-one prediction.

positive:
Existing research on continual learning of a sequence of tasks focused on
dealing with catastrophic forgetting, where the tasks are assumed to be
dissimilar and have little shared knowledge. Some work has also been done to
transfer previously learned knowledge to the new task when the tasks are
similar and have shared knowledge. To the best of our knowledge, no technique
has been proposed to learn a sequence of mixed similar and dissimilar tasks
that can deal with forgetting and also transfer knowledge forward and backward.
This paper proposes such a technique to learn both types of tasks in the same
network. For dissimilar tasks, the algorithm focuses on dealing with
forgetting, and for similar tasks, the algorithm focuses on selectively
transferring the knowledge learned from some similar previous tasks to improve
the new task learning. Additionally, the algorithm automatically detects
whether a new task is similar to any previous tasks. Empirical evaluation using
sequences of mixed tasks demonstrates the effectiveness of the proposed model.

positive:
Most recently proposed methods for Neural Program Induction work under the
assumption of having a large set of input/output (I/O) examples for learning
any underlying input-output mapping. This paper aims to address the problem of
data and computation efficiency of program induction by leveraging information
from related tasks. Specifically, we propose two approaches for cross-task
knowledge transfer to improve program induction in limited-data scenarios. In
our first proposal, portfolio adaptation, a set of induction models is
pretrained on a set of related tasks, and the best model is adapted towards the
new task using transfer learning. In our second approach, meta program
induction, a $k$-shot learning approach is used to make a model generalize to
new tasks without additional training. To test the efficacy of our methods, we
constructed a new benchmark of programs written in the Karel programming
language. Using an extensive experimental evaluation on the Karel benchmark, we
demonstrate that our proposals dramatically outperform the baseline induction
method that does not use knowledge transfer. We also analyze the relative
performance of the two approaches and study conditions in which they perform
best. In particular, meta induction outperforms all existing approaches under
extreme data sparsity (when a very small number of examples are available),
i.e., fewer than ten. As the number of available I/O examples increase (i.e. a
thousand or more), portfolio adapted program induction becomes the best
approach. For intermediate data sizes, we demonstrate that the combined method
of adapted meta program induction has the strongest performance.

positive:
Training deep learning models with limited labelled data is an attractive
scenario for many NLP tasks, including document classification. While with the
recent emergence of BERT, deep learning language models can achieve reasonably
good performance in document classification with few labelled instances, there
is a lack of evidence in the utility of applying BERT-like models on long
document classification. This work introduces a long-text-specific model -- the
Hierarchical BERT Model (HBM) -- that learns sentence-level features of the
text and works well in scenarios with limited labelled data. Various evaluation
experiments have demonstrated that HBM can achieve higher performance in
document classification than the previous state-of-the-art methods with only 50
to 200 labelled instances, especially when documents are long. Also, as an
extra benefit of HBM, the salient sentences identified by learned HBM are
useful as explanations for labelling documents based on a user study.

positive:
Given new tasks with very little data$-$such as new classes in a
classification problem or a domain shift in the input$-$performance of modern
vision systems degrades remarkably quickly. In this work, we illustrate how the
neural network representations which underpin modern vision systems are subject
to supervision collapse, whereby they lose any information that is not
necessary for performing the training task, including information that may be
necessary for transfer to new tasks or domains. We then propose two methods to
mitigate this problem. First, we employ self-supervised learning to encourage
general-purpose features that transfer better. Second, we propose a novel
Transformer based neural network architecture called CrossTransformers, which
can take a small number of labeled images and an unlabeled query, find coarse
spatial correspondence between the query and the labeled images, and then infer
class membership by computing distances between spatially-corresponding
features. The result is a classifier that is more robust to task and domain
shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a
recent dataset for evaluating transfer from ImageNet to many other vision
datasets.

positive:
Imitation learning has been commonly applied to solve different tasks in
isolation. This usually requires either careful feature engineering, or a
significant number of samples. This is far from what we desire: ideally, robots
should be able to learn from very few demonstrations of any given task, and
instantly generalize to new situations of the same task, without requiring
task-specific engineering. In this paper, we propose a meta-learning framework
for achieving such capability, which we call one-shot imitation learning.
  Specifically, we consider the setting where there is a very large set of
tasks, and each task has many instantiations. For example, a task could be to
stack all blocks on a table into a single tower, another task could be to place
all blocks on a table into two-block towers, etc. In each case, different
instances of the task would consist of different sets of blocks with different
initial states. At training time, our algorithm is presented with pairs of
demonstrations for a subset of all tasks. A neural net is trained that takes as
input one demonstration and the current state (which initially is the initial
state of the other demonstration of the pair), and outputs an action with the
goal that the resulting sequence of states and actions matches as closely as
possible with the second demonstration. At test time, a demonstration of a
single instance of a new task is presented, and the neural net is expected to
perform well on new instances of this new task. The use of soft attention
allows the model to generalize to conditions and tasks unseen in the training
data. We anticipate that by training this model on a much greater variety of
tasks and settings, we will obtain a general system that can turn any
demonstrations into robust policies that can accomplish an overwhelming variety
of tasks.
  Videos available at https://bit.ly/nips2017-oneshot .

positive:
The performance of deep neural networks improves with more annotated data.
The problem is that the budget for annotation is limited. One solution to this
is active learning, where a model asks human to annotate data that it perceived
as uncertain. A variety of recent methods have been proposed to apply active
learning to deep networks but most of them are either designed specific for
their target tasks or computationally inefficient for large networks. In this
paper, we propose a novel active learning method that is simple but
task-agnostic, and works efficiently with the deep networks. We attach a small
parametric module, named "loss prediction module," to a target network, and
learn it to predict target losses of unlabeled inputs. Then, this module can
suggest data that the target model is likely to produce a wrong prediction.
This method is task-agnostic as networks are learned from a single loss
regardless of target tasks. We rigorously validate our method through image
classification, object detection, and human pose estimation, with the recent
network architectures. The results demonstrate that our method consistently
outperforms the previous methods over the tasks.

positive:
Text alignment finds application in tasks such as citation recommendation and
plagiarism detection. Existing alignment methods operate at a single,
predefined level and cannot learn to align texts at, for example, sentence and
document levels. We propose a new learning approach that equips previously
established hierarchical attention encoders for representing documents with a
cross-document attention component, enabling structural comparisons across
different levels (document-to-document and sentence-to-document). Our component
is weakly supervised from document pairs and can align at multiple levels. Our
evaluation on predicting document-to-document relationships and
sentence-to-document relationships on the tasks of citation recommendation and
plagiarism detection shows that our approach outperforms previously established
hierarchical, attention encoders based on recurrent and transformer
contextualization that are unaware of structural correspondence between
documents.

positive:
Most automation in machine learning focuses on model selection and hyper
parameter tuning, and many overlook the challenge of automatically defining
predictive tasks. We still heavily rely on human experts to define prediction
tasks, and generate labels by aggregating raw data. In this paper, we tackle
the challenge of defining useful prediction problems on event-driven
time-series data. We introduce MLFriend to address this challenge. MLFriend
first generates all possible prediction tasks under a predefined space, then
interacts with a data scientist to learn the context of the data and recommend
good prediction tasks from all the tasks in the space. We evaluate our system
on three different datasets and generate a total of 2885 prediction tasks and
solve them. Out of these 722 were deemed useful by expert data scientists. We
also show that an automatic prediction task discovery system is able to
identify top 10 tasks that a user may like within a batch of 100 tasks.

positive:
State-of-the-art systems in deep question answering proceed as follows: (1)
an initial document retrieval selects relevant documents, which (2) are then
processed by a neural network in order to extract the final answer. Yet the
exact interplay between both components is poorly understood, especially
concerning the number of candidate documents that should be retrieved. We show
that choosing a static number of documents -- as used in prior research --
suffers from a noise-information trade-off and yields suboptimal results. As a
remedy, we propose an adaptive document retrieval model. This learns the
optimal candidate number for document retrieval, conditional on the size of the
corpus and the query. We report extensive experimental results showing that our
adaptive approach outperforms state-of-the-art methods on multiple benchmark
datasets, as well as in the context of corpora with variable sizes.

positive:
Building upon recent Deep Neural Network architectures, current approaches
lying in the intersection of computer vision and natural language processing
have achieved unprecedented breakthroughs in tasks like automatic captioning or
image retrieval. Most of these learning methods, though, rely on large training
sets of images associated with human annotations that specifically describe the
visual content. In this paper we propose to go a step further and explore the
more complex cases where textual descriptions are loosely related to the
images. We focus on the particular domain of News articles in which the textual
content often expresses connotative and ambiguous relations that are only
suggested but not directly inferred from images. We introduce new deep learning
methods that address source detection, popularity prediction, article
illustration and geolocation of articles. An adaptive CNN architecture is
proposed, that shares most of the structure for all the tasks, and is suitable
for multitask and transfer learning. Deep Canonical Correlation Analysis is
deployed for article illustration, and a new loss function based on Great
Circle Distance is proposed for geolocation. Furthermore, we present
BreakingNews, a novel dataset with approximately 100K news articles including
images, text and captions, and enriched with heterogeneous meta-data (such as
GPS coordinates and popularity metrics). We show this dataset to be appropriate
to explore all aforementioned problems, for which we provide a baseline
performance using various Deep Learning architectures, and different
representations of the textual and visual features. We report very promising
results and bring to light several limitations of current state-of-the-art in
this kind of domain, which we hope will help spur progress in the field.

positive:
Over the past decade, convolutional neural networks (CNN) have shown very
competitive performance in medical image analysis tasks, such as disease
classification, tumor segmentation, and lesion detection. CNN has great
advantages in extracting local features of images. However, due to the locality
of convolution operation, it can not deal with long-range relationships well.
Recently, transformers have been applied to computer vision and achieved
remarkable success in large-scale datasets. Compared with natural images,
multi-modal medical images have explicit and important long-range dependencies,
and effective multi-modal fusion strategies can greatly improve the performance
of deep models. This prompts us to study transformer-based structures and apply
them to multi-modal medical images. Existing transformer-based network
architectures require large-scale datasets to achieve better performance.
However, medical imaging datasets are relatively small, which makes it
difficult to apply pure transformers to medical image analysis. Therefore, we
propose TransMed for multi-modal medical image classification. TransMed
combines the advantages of CNN and transformer to efficiently extract low-level
features of images and establish long-range dependencies between modalities. We
evaluated our model for the challenging problem of preoperative diagnosis of
parotid gland tumors, and the experimental results show the advantages of our
proposed method. We argue that the combination of CNN and transformer has
tremendous potential in a large number of medical image analysis tasks. To our
best knowledge, this is the first work to apply transformers to medical image
classification.

positive:
This paper develops a model that addresses sentence embedding, a hot topic in
current natural language processing research, using recurrent neural networks
with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long
term memory, the LSTM-RNN accumulates increasingly richer information as it
goes through the sentence, and when it reaches the last word, the hidden layer
of the network provides a semantic representation of the whole sentence. In
this paper, the LSTM-RNN is trained in a weakly supervised manner on user
click-through data logged by a commercial web search engine. Visualization and
analysis are performed to understand how the embedding process works. The model
is found to automatically attenuate the unimportant words and detects the
salient keywords in the sentence. Furthermore, these detected keywords are
found to automatically activate different cells of the LSTM-RNN, where words
belonging to a similar topic activate the same cell. As a semantic
representation of the sentence, the embedding vector can be used in many
different applications. These automatic keyword detection and topic allocation
abilities enabled by the LSTM-RNN allow the network to perform document
retrieval, a difficult language processing task, where the similarity between
the query and documents can be measured by the distance between their
corresponding sentence embedding vectors computed by the LSTM-RNN. On a web
search task, the LSTM-RNN embedding is shown to significantly outperform
several existing state of the art methods. We emphasize that the proposed model
generates sentence embedding vectors that are specially useful for web document
retrieval tasks. A comparison with a well known general sentence embedding
method, the Paragraph Vector, is performed. The results show that the proposed
method in this paper significantly outperforms it for web document retrieval
task.

positive:
This paper presents a meta-learning framework for few-shots One-Class
Classification (OCC) at test-time, a setting where labeled examples are only
available for the positive class, and no supervision is given for the negative
example. We consider that we have a set of `one-class classification'
objective-tasks with only a small set of positive examples available for each
task, and a set of training tasks with full supervision (i.e. highly imbalanced
classification). We propose an approach using order-equivariant networks to
learn a 'meta' binary-classifier. The model will take as input an example to
classify from a given task, as well as the corresponding supervised set of
positive examples for this OCC task. Thus, the output of the model will be
'conditioned' on the available positive example of a given task, allowing to
predict on new tasks and new examples without labeled negative examples. In
this paper, we are motivated by an astronomy application. Our goal is to
identify if stars belong to a specific stellar group (the 'one-class' for a
given task), called \textit{stellar streams}, where each stellar stream is a
different OCC-task. We show that our method transfers well on unseen (test)
synthetic streams, and outperforms the baselines even though it is not
retrained and accesses a much smaller part of the data per task to predict
(only positive supervision). We see however that it doesn't transfer as well on
the real stream GD-1. This could come from intrinsic differences from the
synthetic and real stream, highlighting the need for consistency in the
'nature' of the task for this method. However, light fine-tuning improve
performances and outperform our baselines. Our experiments show encouraging
results to further explore meta-learning methods for OCC tasks.

positive:
Lifelong learning has recently attracted attention in building machine
learning systems that continually accumulate and transfer knowledge to help
future learning. Unsupervised topic modeling has been popularly used to
discover topics from document collections. However, the application of topic
modeling is challenging due to data sparsity, e.g., in a small collection of
(short) documents and thus, generate incoherent topics and sub-optimal document
representations. To address the problem, we propose a lifelong learning
framework for neural topic modeling that can continuously process streams of
document collections, accumulate topics and guide future topic modeling tasks
by knowledge transfer from several sources to better deal with the sparse data.
In the lifelong process, we particularly investigate jointly: (1) sharing
generative homologies (latent topics) over lifetime to transfer prior
knowledge, and (2) minimizing catastrophic forgetting to retain the past
learning via novel selective data augmentation, co-training and topic
regularization approaches. Given a stream of document collections, we apply the
proposed Lifelong Neural Topic Modeling (LNTM) framework in modeling three
sparse document collections as future tasks and demonstrate improved
performance quantified by perplexity, topic coherence and information retrieval
task.

positive:
A general-purpose robot should be able to master a wide range of tasks and
quickly learn a novel one by leveraging past experiences. One-shot imitation
learning (OSIL) approaches this goal by training an agent with (pairs of)
expert demonstrations, such that at test time, it can directly execute a new
task from just one demonstration. However, so far this framework has been
limited to training on many variations of one task, and testing on other unseen
but similar variations of the same task. In this work, we push for a higher
level of generalization ability by investigating a more ambitious multi-task
setup. We introduce a diverse suite of vision-based robot manipulation tasks,
consisting of 7 tasks, a total of 61 variations, and a continuum of instances
within each variation. For consistency and comparison purposes, we first train
and evaluate single-task agents (as done in prior few-shot imitation work). We
then study the multi-task setting, where multi-task training is followed by (i)
one-shot imitation on variations within the training tasks, (ii) one-shot
imitation on new tasks, and (iii) fine-tuning on new tasks. Prior
state-of-the-art, while performing well within some single tasks, struggles in
these harder multi-task settings. To address these limitations, we propose
MOSAIC (Multi-task One-Shot Imitation with self-Attention and Contrastive
learning), which integrates a self-attention model architecture and a temporal
contrastive module to enable better task disambiguation and more robust
representation learning. Our experiments show that MOSAIC outperforms prior
state of the art in learning efficiency, final performance, and learns a
multi-task policy with promising generalization ability via fine-tuning on
novel tasks.

positive:
Transfer learning has been developed to improve the performances of different
but related tasks in machine learning. However, such processes become less
efficient with the increase of the size of training data and the number of
tasks. Moreover, privacy can be violated as some tasks may contain sensitive
and private data, which are communicated between nodes and tasks. We propose a
consensus-based distributed transfer learning framework, where several tasks
aim to find the best linear support vector machine (SVM) classifiers in a
distributed network. With alternating direction method of multipliers, tasks
can achieve better classification accuracies more efficiently and privately, as
each node and each task train with their own data, and only decision variables
are transferred between different tasks and nodes. Numerical experiments on
MNIST datasets show that the knowledge transferred from the source tasks can be
used to decrease the risks of the target tasks that lack training data or have
unbalanced training labels. We show that the risks of the target tasks in the
nodes without the data of the source tasks can also be reduced using the
information transferred from the nodes who contain the data of the source
tasks. We also show that the target tasks can enter and leave in real-time
without rerunning the whole algorithm.

positive:
Many structured prediction tasks in machine vision have a collection of
acceptable answers, instead of one definitive ground truth answer. Segmentation
of images, for example, is subject to human labeling bias. Similarly, there are
multiple possible pixel values that could plausibly complete occluded image
regions. State-of-the art supervised learning methods are typically optimized
to make a single test-time prediction for each query, failing to find other
modes in the output space. Existing methods that allow for sampling often
sacrifice speed or accuracy.
  We introduce a simple method for training a neural network, which enables
diverse structured predictions to be made for each test-time query. For a
single input, we learn to predict a range of possible answers. We compare
favorably to methods that seek diversity through an ensemble of networks. Such
stochastic multiple choice learning faces mode collapse, where one or more
ensemble members fail to receive any training signal. Our best performing
solution can be deployed for various tasks, and just involves small
modifications to the existing single-mode architecture, loss function, and
training regime. We demonstrate that our method results in quantitative
improvements across three challenging tasks: 2D image completion, 3D volume
estimation, and flow prediction.

negative:
Transferring knowledge from prior source tasks in solving a new target task
can be useful in several learning applications. The application of transfer
poses two serious challenges which have not been adequately addressed. First,
the agent should be able to avoid negative transfer, which happens when the
transfer hampers or slows down the learning instead of helping it. Second, the
agent should be able to selectively transfer, which is the ability to select
and transfer from different and multiple source tasks for different parts of
the state space of the target task. We propose A2T (Attend, Adapt and
Transfer), an attentive deep architecture which adapts and transfers from these
source tasks. Our model is generic enough to effect transfer of either policies
or value functions. Empirical evaluations on different learning algorithms show
that A2T is an effective architecture for transfer by being able to avoid
negative transfer while transferring selectively from multiple source tasks in
the same domain.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
Modern deep learning based classifiers show very high accuracy on test data
but this does not provide sufficient guarantees for safe deployment, especially
in high-stake AI applications such as medical diagnosis. Usually, predictions
are obtained without a reliable uncertainty estimate or a formal guarantee.
Conformal prediction (CP) addresses these issues by using the classifier's
predictions, e.g., its probability estimates, to predict confidence sets
containing the true class with a user-specified probability. However, using CP
as a separate processing step after training prevents the underlying model from
adapting to the prediction of confidence sets. Thus, this paper explores
strategies to differentiate through CP during training with the goal of
training model with the conformal wrapper end-to-end. In our approach,
conformal training (ConfTr), we specifically "simulate" conformalization on
mini-batches during training. Compared to standard training, ConfTr reduces the
average confidence set size (inefficiency) of state-of-the-art CP methods
applied after training. Moreover, it allows to "shape" the confidence sets
predicted at test time, which is difficult for standard CP. On experiments with
several datasets, we show ConfTr can influence how inefficiency is distributed
across classes, or guide the composition of confidence sets in terms of the
included classes, while retaining the guarantees offered by CP.

negative:
Convolutional Neural Networks (CNNs) have recently emerged as the dominant
model in computer vision. If provided with enough training data, they predict
almost any visual quantity. In a discrete setting, such as classification, CNNs
are not only able to predict a label but often predict a confidence in the form
of a probability distribution over the output space. In continuous regression
tasks, such a probability estimate is often lacking. We present a regression
framework which models the output distribution of neural networks. This output
distribution allows us to infer the most likely labeling following a set of
physical or modeling constraints. These constraints capture the intricate
interplay between different input and output variables, and complement the
output of a CNN. However, they may not hold everywhere. Our setup further
allows to learn a confidence with which a constraint holds, in the form of a
distribution of the constrain satisfaction. We evaluate our approach on the
problem of intrinsic image decomposition, and show that constrained structured
regression significantly increases the state-of-the-art.

negative:
Although recent multi-task learning methods have shown to be effective in
improving the generalization of deep neural networks, they should be used with
caution for safety-critical applications, such as clinical risk prediction.
This is because even if they achieve improved task-average performance, they
may still yield degraded performance on individual tasks, which may be critical
(e.g., prediction of mortality risk). Existing asymmetric multi-task learning
methods tackle this negative transfer problem by performing knowledge transfer
from tasks with low loss to tasks with high loss. However, using loss as a
measure of reliability is risky since it could be a result of overfitting. In
the case of time-series prediction tasks, knowledge learned for one task (e.g.,
predicting the sepsis onset) at a specific timestep may be useful for learning
another task (e.g., prediction of mortality) at a later timestep, but lack of
loss at each timestep makes it difficult to measure the reliability at each
timestep. To capture such dynamically changing asymmetric relationships between
tasks in time-series data, we propose a novel temporal asymmetric multi-task
learning model that performs knowledge transfer from certain tasks/timesteps to
relevant uncertain tasks, based on feature-level uncertainty. We validate our
model on multiple clinical risk prediction tasks against various deep learning
models for time-series prediction, which our model significantly outperforms,
without any sign of negative transfer. Further qualitative analysis of learned
knowledge graphs by clinicians shows that they are helpful in analyzing the
predictions of the model. Our final code is available at
https://github.com/anhtuan5696/TPAMTL.

negative:
Convolutional Neural Networks experience catastrophic forgetting when
optimized on a sequence of learning problems: as they meet the objective of the
current training examples, their performance on previous tasks drops
drastically. In this work, we introduce a novel framework to tackle this
problem with conditional computation. We equip each convolutional layer with
task-specific gating modules, selecting which filters to apply on the given
input. This way, we achieve two appealing properties. Firstly, the execution
patterns of the gates allow to identify and protect important filters, ensuring
no loss in the performance of the model for previously learned tasks. Secondly,
by using a sparsity objective, we can promote the selection of a limited set of
kernels, allowing to retain sufficient model capacity to digest new
tasks.Existing solutions require, at test time, awareness of the task to which
each example belongs to. This knowledge, however, may not be available in many
practical scenarios. Therefore, we additionally introduce a task classifier
that predicts the task label of each example, to deal with settings in which a
task oracle is not available. We validate our proposal on four continual
learning datasets. Results show that our model consistently outperforms
existing methods both in the presence and the absence of a task oracle.
Notably, on Split SVHN and Imagenet-50 datasets, our model yields up to 23.98%
and 17.42% improvement in accuracy w.r.t. competing methods.

negative:
There are relatively few works dealing with conformal prediction for
multi-task learning issues, and this is particularly true for multi-target
regression. This paper focuses on the problem of providing valid (i.e.,
frequency calibrated) multi-variate predictions. To do so, we propose to use
copula functions applied to deep neural networks for inductive conformal
prediction. We show that the proposed method ensures efficiency and validity
for multi-target regression problems on various data sets.

negative:
Multitask learning can be effective when features useful in one task are also
useful for other tasks, and the group lasso is a standard method for selecting
a common subset of features. In this paper, we are interested in a less
restrictive form of multitask learning, wherein (1) the available features can
be organized into subsets according to a notion of similarity and (2) features
useful in one task are similar, but not necessarily identical, to the features
best suited for other tasks. The main contribution of this paper is a new
procedure called Sparse Overlapping Sets (SOS) lasso, a convex optimization
that automatically selects similar features for related learning tasks. Error
bounds are derived for SOSlasso and its consistency is established for squared
error loss. In particular, SOSlasso is motivated by multi- subject fMRI studies
in which functional activity is classified using brain voxels as features.
Experiments with real and synthetic data demonstrate the advantages of SOSlasso
compared to the lasso and group lasso.

negative:
Conformal predictors are machine learning algorithms that output prediction
sets that have a guarantee of marginal validity for finite samples with minimal
distributional assumptions. This is a property that makes conformal predictors
useful for machine learning tasks where we require reliable predictions. It
would also be desirable to achieve conditional validity in the same setting, in
the sense that validity of the prediction intervals remains valid regardless of
conditioning on any particular property of the object of the prediction.
Unfortunately, it has been shown that such conditional validity is impossible
to guarantee for non-trivial prediction problems for finite samples. In this
article, instead of trying to achieve a strong conditional validity result, the
weaker goal of achieving an approximation to conditional validity is
considered. A new algorithm is introduced to do this by iteratively adjusting a
conformity measure to deviations from object conditional validity measured in
the training data. Along with some theoretical results, experimental results
are provided for three data sets that demonstrate (1) in real world machine
learning tasks, lack of conditional validity is a measurable problem and (2)
that the proposed algorithm is effective at alleviating this problem.

negative:
In class-incremental learning, an agent with limited resources needs to learn
a sequence of classification tasks, forming an ever growing classification
problem, with the constraint of not being able to access data from previous
tasks. The main difference with task-incremental learning, where a task-ID is
available at inference time, is that the learner also needs to perform
cross-task discrimination, i.e. distinguish between classes that have not been
seen together. Approaches to tackle this problem are numerous and mostly make
use of an external memory (buffer) of non-negligible size. In this paper, we
ablate the learning of cross-task features and study its influence on the
performance of basic replay strategies used for class-IL. We also define a new
forgetting measure for class-incremental learning, and see that forgetting is
not the principal cause of low performance. Our experimental results show that
future algorithms for class-incremental learning should not only prevent
forgetting, but also aim to improve the quality of the cross-task features, and
the knowledge transfer between tasks. This is especially important when tasks
contain limited amount of data.

negative:
Convolutional Neural Networks (CNNs) have shown to be powerful medical image
segmentation models. In this study, we address some of the main unresolved
issues regarding these models. Specifically, training of these models on small
medical image datasets is still challenging, with many studies promoting
techniques such as transfer learning. Moreover, these models are infamous for
producing over-confident predictions and for failing silently when presented
with out-of-distribution (OOD) data at test time. In this paper, we advocate
for multi-task learning, i.e., training a single model on several different
datasets, spanning several different organs of interest and different imaging
modalities. We show that not only a single CNN learns to automatically
recognize the context and accurately segment the organ of interest in each
context, but also that such a joint model often has more accurate and
better-calibrated predictions than dedicated models trained separately on each
dataset. Our experiments show that multi-task learning can outperform transfer
learning in medical image segmentation tasks. For detecting OOD data, we
propose a method based on spectral analysis of CNN feature maps. We show that
different datasets, representing different imaging modalities and/or different
organs of interest, have distinct spectral signatures, which can be used to
identify whether or not a test image is similar to the images used to train a
model. We show that this approach is far more accurate than OOD detection based
on prediction uncertainty. The methods proposed in this paper contribute
significantly to improving the accuracy and reliability of CNN-based medical
image segmentation models.

negative:
The task of multi-label image classification is to recognize all the object
labels presented in an image. Though advancing for years, small objects,
similar objects and objects with high conditional probability are still the
main bottlenecks of previous convolutional neural network(CNN) based models,
limited by convolutional kernels' representational capacity. Recent vision
transformer networks utilize the self-attention mechanism to extract the
feature of pixel granularity, which expresses richer local semantic
information, while is insufficient for mining global spatial dependence. In
this paper, we point out the three crucial problems that CNN-based methods
encounter and explore the possibility of conducting specific transformer
modules to settle them. We put forward a Multi-label Transformer
architecture(MlTr) constructed with windows partitioning, in-window pixel
attention, cross-window attention, particularly improving the performance of
multi-label image classification tasks. The proposed MlTr shows
state-of-the-art results on various prevalent multi-label datasets such as
MS-COCO, Pascal-VOC, and NUS-WIDE with 88.5%, 95.8%, and 65.5% respectively.
The code will be available soon at https://github.com/starmemda/MlTr/

negative:
We consider the problem of transfer learning -- gaining knowledge from one
source task and applying it to a different but related target task. A
fundamental question in transfer learning is whether combining the data of both
tasks works better than using only the target task's data (equivalently,
whether a "positive information transfer" happens). We study this question
formally in a linear regression setting where a two-layer linear neural network
estimator combines both tasks' data. The estimator uses a shared parameter
vector for both tasks and exhibits positive or negative information transfer by
varying dataset characteristics.
  We characterize the precise asymptotic limit of the prediction risk of the
above estimator when the sample sizes increase with the feature dimension
proportionally at fixed ratios. We also show that the asymptotic limit is
sufficiently accurate for finite dimensions. Then, we provide the exact
condition to determine positive (and negative) information transfer in a
random-effect model, leading to several theoretical insights. For example, the
risk curve is non-monotone under model shift, thus motivating a transfer
learning procedure that progressively adds data from the source task. We
validate this procedure's efficiency on text classification tasks with a neural
network that applies a shared feature space for both tasks, similar to the
above estimator. The main ingredient of the analysis is finding the
high-dimensional asymptotic limits of various functions involving the sum of
two independent sample covariance matrices with different population covariance
matrices, which may be of independent interest.

negative:
This paper presents an online transfer learning framework for improving
temperature predictions in residential buildings. In transfer learning,
prediction models trained under a set of available data from a target domain
(e.g., house with limited data) can be improved through the use of data
generated from similar source domains (e.g., houses with rich data). Given also
the need for prediction models that can be trained online (e.g., as part of a
model-predictive-control implementation), this paper introduces the generalized
online transfer learning algorithm (GOTL). It employs a weighted combination of
the available predictors (i.e., the target and source predictors) and
guarantees convergence to the best weighted predictor. Furthermore, the use of
Transfer Component Analysis (TCA) allows for using more than a single source
domains, since it may facilitate the fit of a single model on more than one
source domains (houses). This allows GOTL to transfer knowledge from more than
one source domains. We further validate our results through experiments in
climate control for residential buildings and show that GOTL may lead to
non-negligible energy savings for given comfort levels.

negative:
In this work, we focus on the task of generating natural language
descriptions from a structured table of facts containing fields (such as
nationality, occupation, etc) and values (such as Indian, actor, director,
etc). One simple choice is to treat the table as a sequence of fields and
values and then use a standard seq2seq model for this task. However, such a
model is too generic and does not exploit task-specific characteristics. For
example, while generating descriptions from a table, a human would attend to
information at two levels: (i) the fields (macro level) and (ii) the values
within the field (micro level). Further, a human would continue attending to a
field for a few timesteps till all the information from that field has been
rendered and then never return back to this field (because there is nothing
left to say about it). To capture this behavior we use (i) a fused bifocal
attention mechanism which exploits and combines this micro and macro level
information and (ii) a gated orthogonalization mechanism which tries to ensure
that a field is remembered for a few time steps and then forgotten. We
experiment with a recently released dataset which contains fact tables about
people and their corresponding one line biographical descriptions in English.
In addition, we also introduce two similar datasets for French and German. Our
experiments show that the proposed model gives 21% relative improvement over a
recently proposed state of the art method and 10% relative improvement over
basic seq2seq models. The code and the datasets developed as a part of this
work are publicly available.

negative:
Previous researchers have considered sentiment analysis as a document
classification task, in which input documents are classified into predefined
sentiment classes. Although there are sentences in a document that support
important evidences for sentiment analysis and sentences that do not, they have
treated the document as a bag of sentences. In other words, they have not
considered the importance of each sentence in the document. To effectively
determine polarity of a document, each sentence in the document should be dealt
with different degrees of importance. To address this problem, we propose a
document-level sentence classification model based on deep neural networks, in
which the importance degrees of sentences in documents are automatically
determined through gate mechanisms. To verify our new sentiment analysis model,
we conducted experiments using the sentiment datasets in the four different
domains such as movie reviews, hotel reviews, restaurant reviews, and music
reviews. In the experiments, the proposed model outperformed previous
state-of-the-art models that do not consider importance differences of
sentences in a document. The experimental results show that the importance of
sentences should be considered in a document-level sentiment classification
task.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
An accurate abstractive summary of a document should contain all its salient
information and should be logically entailed by the input document. We improve
these important aspects of abstractive summarization via multi-task learning
with the auxiliary tasks of question generation and entailment generation,
where the former teaches the summarization model how to look for salient
questioning-worthy details, and the latter teaches the model how to rewrite a
summary which is a directed-logical subset of the input document. We also
propose novel multi-task architectures with high-level (semantic)
layer-specific sharing across multiple encoder and decoder layers of the three
tasks, as well as soft-sharing mechanisms (and show performance ablations and
analysis examples of each contribution). Overall, we achieve statistically
significant improvements over the state-of-the-art on both the CNN/DailyMail
and Gigaword datasets, as well as on the DUC-2002 transfer setup. We also
present several quantitative and qualitative analysis studies of our model's
learned saliency and entailment skills.

negative:
Black-box machine learning models are now routinely used in high-risk
settings, like medical diagnostics, which demand uncertainty quantification to
avoid consequential model failures. Conformal prediction is a user-friendly
paradigm for creating statistically rigorous uncertainty sets/intervals for the
predictions of such models. Critically, the sets are valid in a
distribution-free sense: they possess explicit, non-asymptotic guarantees even
without distributional assumptions or model assumptions. One can use conformal
prediction with any pre-trained model, such as a neural network, to produce
sets that are guaranteed to contain the ground truth with a user-specified
probability, such as 90%. It is easy-to-understand, easy-to-use, and general,
applying naturally to problems arising in the fields of computer vision,
natural language processing, deep reinforcement learning, and so on.
  This hands-on introduction is aimed to provide the reader a working
understanding of conformal prediction and related distribution-free uncertainty
quantification techniques with one self-contained document. We lead the reader
through practical theory for and examples of conformal prediction and describe
its extensions to complex machine learning tasks involving structured outputs,
distribution shift, time-series, outliers, models that abstain, and more.
Throughout, there are many explanatory illustrations, examples, and code
samples in Python. With each code sample comes a Jupyter notebook implementing
the method on a real-data example; the notebooks can be accessed and easily run
using our codebase.

negative:
In many natural language processing (NLP) tasks, a document is commonly
modeled as a bag of words using the term frequency-inverse document frequency
(TF-IDF) vector. One major shortcoming of the frequency-based TF-IDF feature
vector is that it ignores word orders that carry syntactic and semantic
relationships among the words in a document, and they can be important in some
NLP tasks such as genre classification. This paper proposes a novel distributed
vector representation of a document: a simple recurrent-neural-network language
model (RNN-LM) or a long short-term memory RNN language model (LSTM-LM) is
first created from all documents in a task; some of the LM parameters are then
adapted by each document, and the adapted parameters are vectorized to
represent the document. The new document vectors are labeled as DV-RNN and
DV-LSTM respectively. We believe that our new document vectors can capture some
high-level sequential information in the documents, which other current
document representations fail to capture. The new document vectors were
evaluated in the genre classification of documents in three corpora: the Brown
Corpus, the BNC Baby Corpus and an artificially created Penn Treebank dataset.
Their classification performances are compared with the performance of TF-IDF
vector and the state-of-the-art distributed memory model of paragraph vector
(PV-DM). The results show that DV-LSTM significantly outperforms TF-IDF and
PV-DM in most cases, and combinations of the proposed document vectors with
TF-IDF or PV-DM may further improve performance.

negative:
Different people have different habits of describing their intents in
conversations. Some people tend to deliberate their intents in several
successive utterances, i.e., they use several consistent messages for
readability instead of a long sentence to express their question. This creates
a predicament faced by the application of dialogue systems, especially in
real-world industry scenarios, in which the dialogue system is unsure whether
it should answer the query of user immediately or wait for further
supplementary input. Motivated by such an interesting predicament, we define a
novel Wait-or-Answer task for dialogue systems. We shed light on a new research
topic about how the dialogue system can be more intelligent to behave in this
Wait-or-Answer quandary. Further, we propose a predictive approach named
Predict-then-Decide (PTD) to tackle this Wait-or-Answer task. More
specifically, we take advantage of a decision model to help the dialogue system
decide whether to wait or answer. The decision of decision model is made with
the assistance of two ancillary prediction models: a user prediction and an
agent prediction. The user prediction model tries to predict what the user
would supplement and uses its prediction to persuade the decision model that
the user has some information to add, so the dialogue system should wait. The
agent prediction model tries to predict the answer of the dialogue system and
convince the decision model that it is a superior choice to answer the query of
user immediately since the input of user has come to an end. We conduct our
experiments on two real-life scenarios and three public datasets. Experimental
results on five datasets show our proposed PTD approach significantly
outperforms the existing models in solving this Wait-or-Answer problem.

negative:
The Query Focused Text Summarization (QFTS) task aims at building systems
that generate the summary of the text document(s) based on the given query. A
key challenge in addressing this task is the lack of large labeled data for
training the summarization model. In this paper, we address this challenge by
exploring a series of domain adaptation techniques. Given the recent success of
pre-trained transformer models in a wide range of natural language processing
tasks, we utilize such models to generate abstractive summaries for the QFTS
task for both single-document and multi-document scenarios. For domain
adaptation, we apply a variety of techniques using pre-trained
transformer-based summarization models including transfer learning, weakly
supervised learning, and distant supervision. Extensive experiments on six
datasets show that our proposed approach is very effective in generating
abstractive summaries for the QFTS task while setting a new state-of-the-art
result in several datasets across a set of automatic and human evaluation
metrics.

negative:
Conformal predictors are an important class of algorithms that allow
predictions to be made with a user-defined confidence level. They are able to
do this by outputting prediction sets, rather than simple point predictions.
The conformal predictor is valid in the sense that the accuracy of its
predictions is guaranteed to meet the confidence level, only assuming
exchangeability in the data. Since accuracy is guaranteed, the performance of a
conformal predictor is measured through the efficiency of the prediction sets.
Typically, a conformal predictor is built on an underlying machine learning
algorithm and hence its predictive power is inherited from this algorithm.
However, since the underlying machine learning algorithm is not trained with
the objective of minimizing predictive efficiency it means that the resulting
conformal predictor may be sub-optimal and not aligned sufficiently to this
objective. Hence, in this study we consider an approach to train the conformal
predictor directly with maximum predictive efficiency as the optimization
objective, and we focus specifically on the inductive conformal predictor for
classification. To do this, the conformal predictor is approximated by a
differentiable objective function and gradient descent used to optimize it. The
resulting parameter estimates are then passed to a proper inductive conformal
predictor to give valid prediction sets. We test the method on several real
world data sets and find that the method is promising and in most cases gives
improved predictive efficiency against a baseline conformal predictor.

negative:
Neural machine translation (NMT) has arguably achieved human level parity
when trained and evaluated at the sentence-level. Document-level neural machine
translation has received less attention and lags behind its sentence-level
counterpart. The majority of the proposed document-level approaches investigate
ways of conditioning the model on several source or target sentences to capture
document context. These approaches require training a specialized NMT model
from scratch on parallel document-level corpora. We propose an approach that
doesn't require training a specialized model on parallel document-level corpora
and is applied to a trained sentence-level NMT model at decoding time. We
process the document from left to right multiple times and self-train the
sentence-level model on pairs of source sentences and generated translations.
Our approach reinforces the choices made by the model, thus making it more
likely that the same choices will be made in other sentences in the document.
We evaluate our approach on three document-level datasets: NIST
Chinese-English, WMT'19 Chinese-English and OpenSubtitles English-Russian. We
demonstrate that our approach has higher BLEU score and higher human preference
than the baseline. Qualitative analysis of our approach shows that choices made
by model are consistent across the document.

negative:
Conformal Prediction is a framework that produces prediction intervals based
on the output from a machine learning algorithm. In this paper we explore the
case when training data is made up of multiple parts available in different
sources that cannot be pooled. We here consider the regression case and propose
a method where a conformal predictor is trained on each data source
independently, and where the prediction intervals are then combined into a
single interval. We call the approach Non-Disclosed Conformal Prediction
(NDCP), and we evaluate it on a regression dataset from the UCI machine
learning repository using support vector regression as the underlying machine
learning algorithm, with varying number of data sources and sizes. The results
show that the proposed method produces conservatively valid prediction
intervals, and while we cannot retain the same efficiency as when all data is
used, efficiency is improved through the proposed approach as compared to
predicting using a single arbitrarily chosen source.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
We first exhibit a multimodal image registration task, for which a neural
network trained on a dataset with noisy labels reaches almost perfect accuracy,
far beyond noise variance. This surprising auto-denoising phenomenon can be
explained as a noise averaging effect over the labels of similar input
examples. This effect theoretically grows with the number of similar examples;
the question is then to define and estimate the similarity of examples.
  We express a proper definition of similarity, from the neural network
perspective, i.e. we quantify how undissociable two inputs $A$ and $B$ are,
taking a machine learning viewpoint: how much a parameter variation designed to
change the output for $A$ would impact the output for $B$ as well?
  We study the mathematical properties of this similarity measure, and show how
to use it on a trained network to estimate sample density, in low complexity,
enabling new types of statistical analysis for neural networks. We analyze data
by retrieving samples perceived as similar by the network, and are able to
quantify the denoising effect without requiring true labels. We also propose,
during training, to enforce that examples known to be similar should also be
seen as similar by the network, and notice speed-up training effects for
certain datasets.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
Novel contexts may often arise in complex querying scenarios such as in
evidence-based medicine (EBM) involving biomedical literature, that may not
explicitly refer to entities or canonical concept forms occurring in any fact-
or rule-based knowledge source such as an ontology like the UMLS. Moreover,
hidden associations between candidate concepts meaningful in the current
context, may not exist within a single document, but within the collection, via
alternate lexical forms. Therefore, inspired by the recent success of
sequence-to-sequence neural models in delivering the state-of-the-art in a wide
range of NLP tasks, we develop a novel sequence-to-set framework with neural
attention for learning document representations that can effect term transfer
within the corpus, for semantically tagging a large collection of documents. We
demonstrate that our proposed method can be effective in both a supervised
multi-label classification setup for text categorization, as well as in a
unique unsupervised setting with no human-annotated document labels that uses
no external knowledge resources and only corpus-derived term statistics to
drive the training. Further, we show that semi-supervised training using our
architecture on large amounts of unlabeled data can augment performance on the
text categorization task when limited labeled data is available. Our approach
to generate document encodings employing our sequence-to-set models for
inference of semantic tags, gives to the best of our knowledge, the
state-of-the-art for both, the unsupervised query expansion task for the TREC
CDS 2016 challenge dataset when evaluated on an Okapi BM25--based document
retrieval system; and also over the MLTM baseline (Soleimani et al, 2016), for
both supervised and semi-supervised multi-label prediction tasks on the
del.icio.us and Ohsumed datasets. We will make our code and data publicly
available.

negative:
While the use of deep learning in drug discovery is gaining increasing
attention, the lack of methods to compute reliable errors in prediction for
Neural Networks prevents their application to guide decision making in domains
where identifying unreliable predictions is essential, e.g. precision medicine.
Here, we present a framework to compute reliable errors in prediction for
Neural Networks using Test-Time Dropout and Conformal Prediction. Specifically,
the algorithm consists of training a single Neural Network using dropout, and
then applying it N times to both the validation and test sets, also employing
dropout in this step. Therefore, for each instance in the validation and test
sets an ensemble of predictions were generated. The residuals and absolute
errors in prediction for the validation set were then used to compute
prediction errors for test set instances using Conformal Prediction. We show
using 24 bioactivity data sets from ChEMBL 23 that dropout Conformal Predictors
are valid (i.e., the fraction of instances whose true value lies within the
predicted interval strongly correlates with the confidence level) and
efficient, as the predicted confidence intervals span a narrower set of values
than those computed with Conformal Predictors generated using Random Forest
(RF) models. Lastly, we show in retrospective virtual screening experiments
that dropout and RF-based Conformal Predictors lead to comparable retrieval
rates of active compounds. Overall, we propose a computationally efficient
framework (as only N extra forward passes are required in addition to training
a single network) to harness Test-Time Dropout and the Conformal Prediction
framework, and to thereby generate reliable prediction errors for deep Neural
Networks.

negative:
Most classifiers operate by selecting the maximum of an estimate of the
conditional distribution $p(y|x)$ where $x$ stands for the features of the
instance to be classified and $y$ denotes its label. This often results in a
{\em hubristic bias}: overconfidence in the assignment of a definite label.
Usually, the observations are concentrated on a small volume but the classifier
provides definite predictions for the entire space. We propose constructing
conformal prediction sets which contain a set of labels rather than a single
label. These conformal prediction sets contain the true label with probability
$1-\alpha$. Our construction is based on $p(x|y)$ rather than $p(y|x)$ which
results in a classifier that is very cautious: it outputs the null set ---
meaning "I don't know" --- when the object does not resemble the training
examples. An important property of our approach is that adversarial attacks are
likely to be predicted as the null set or would also include the true label. We
demonstrate the performance on the ImageNet ILSVRC dataset and the CelebA and
IMDB-Wiki facial datasets using high dimensional features obtained from state
of the art convolutional neural networks.

negative:
Standard neural machine translation (NMT) is on the assumption of
document-level context independent. Most existing document-level NMT methods
are satisfied with a smattering sense of brief document-level information,
while this work focuses on exploiting detailed document-level context in terms
of multiple forms of document embeddings, which is capable of sufficiently
modeling deeper and richer document-level context. The proposed document-aware
NMT is implemented to enhance the Transformer baseline by introducing both
global and local document-level clues on the source end. Experiments show that
the proposed method significantly improves the translation performance over
strong baselines and other related studies.

negative:
Sharing information between multiple tasks enables algorithms to achieve good
generalization performance even from small amounts of training data. However,
in a realistic scenario of multi-task learning not all tasks are equally
related to each other, hence it could be advantageous to transfer information
only between the most related tasks. In this work we propose an approach that
processes multiple tasks in a sequence with sharing between subsequent tasks
instead of solving all tasks jointly. Subsequently, we address the question of
curriculum learning of tasks, i.e. finding the best order of tasks to be
learned. Our approach is based on a generalization bound criterion for choosing
the task order that optimizes the average expected classification performance
over all tasks. Our experimental results show that learning multiple related
tasks sequentially can be more effective than learning them jointly, the order
in which tasks are being solved affects the overall performance, and that our
model is able to automatically discover the favourable order of tasks.

negative:
We investigate whether and where multi-task learning (MTL) can improve
performance on NLP problems related to argumentation mining (AM), in particular
argument component identification. Our results show that MTL performs
particularly well (and better than single-task learning) when little training
data is available for the main task, a common scenario in AM. Our findings
challenge previous assumptions that conceptualizations across AM datasets are
divergent and that MTL is difficult for semantic or higher-level tasks.

negative:
The machine learning literature contains several constructions for prediction
intervals that are intuitively reasonable but ultimately ad-hoc in that they do
not come with provable performance guarantees. We present methods from the
statistics literature that can be used efficiently with neural networks under
minimal assumptions with guaranteed performance. We propose a neural network
that outputs three values instead of a single point estimate and optimizes a
loss function motivated by the standard quantile regression loss. We provide
two prediction interval methods with finite sample coverage guarantees solely
under the assumption that the observations are independent and identically
distributed. The first method leverages the conformal inference framework and
provides average coverage. The second method provides a new, stronger guarantee
by conditioning on the observed data. Lastly, our loss function does not
compromise the predictive accuracy of the network like other prediction
interval methods. We demonstrate the ease of use of our procedures as well as
its improvements over other methods on both simulated and real data. As most
deep networks can easily be modified by our method to output predictions with
valid prediction intervals, its use should become standard practice, much like
reporting standard errors along with mean estimates.

negative:
In the era of pre-trained language models, Transformers are the de facto
choice of model architectures. While recent research has shown promise in
entirely convolutional, or CNN, architectures, they have not been explored
using the pre-train-fine-tune paradigm. In the context of language models, are
convolutional models competitive to Transformers when pre-trained? This paper
investigates this research question and presents several interesting findings.
Across an extensive set of experiments on 8 datasets/tasks, we find that
CNN-based pre-trained models are competitive and outperform their Transformer
counterpart in certain scenarios, albeit with caveats. Overall, the findings
outlined in this paper suggest that conflating pre-training and architectural
advances is misguided and that both advances should be considered
independently. We believe our research paves the way for a healthy amount of
optimism in alternative architectures.

negative:
Due to the increasing amount of data on the internet, finding a
highly-informative, low-dimensional representation for text is one of the main
challenges for efficient natural language processing tasks including text
classification. This representation should capture the semantic information of
the text while retaining their relevance level for document classification.
This approach maps the documents with similar topics to a similar space in
vector space representation. To obtain representation for large text, we
propose the utilization of deep Siamese neural networks. To embed document
relevance in topics in the distributed representation, we use a Siamese neural
network to jointly learn document representations. Our Siamese network consists
of two sub-network of multi-layer perceptron. We examine our representation for
the text categorization task on BBC news dataset. The results show that the
proposed representations outperform the conventional and state-of-the-art
representations in the text classification task on this dataset.

negative:
Training recurrent neural networks to model long term dependencies is
difficult. Hence, we propose to use external linguistic knowledge as an
explicit signal to inform the model which memories it should utilize.
Specifically, external knowledge is used to augment a sequence with typed edges
between arbitrarily distant elements, and the resulting graph is decomposed
into directed acyclic subgraphs. We introduce a model that encodes such graphs
as explicit memory in recurrent neural networks, and use it to model
coreference relations in text. We apply our model to several text comprehension
tasks and achieve new state-of-the-art results on all considered benchmarks,
including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out
of the 20 tasks with only 1000 training examples per task. Analysis of the
learned representations further demonstrates the ability of our model to encode
fine-grained entity information across a document.

negative:
A reasonable approach for fact checking a claim involves retrieving
potentially relevant documents from different sources (e.g., news websites,
social media, etc.), determining the stance of each document with respect to
the claim, and finally making a prediction about the claim's factuality by
aggregating the strength of the stances, while taking the reliability of the
source into account. Moreover, a fact checking system should be able to explain
its decision by providing relevant extracts (rationales) from the documents.
Yet, this setup is not directly supported by existing datasets, which treat
fact checking, document retrieval, source credibility, stance detection and
rationale extraction as independent tasks. In this paper, we support the
interdependencies between these tasks as annotations in the same corpus. We
implement this setup on an Arabic fact checking corpus, the first of its kind.

negative:
Explaining the predictions of AI models is paramount in safety-critical
applications, such as in legal or medical domains. One form of explanation for
a prediction is an extractive rationale, i.e., a subset of features of an
instance that lead the model to give its prediction on the instance. Previous
works on generating extractive rationales usually employ a two-phase model: a
selector that selects the most important features (i.e., the rationale)
followed by a predictor that makes the prediction based exclusively on the
selected features. One disadvantage of these works is that the main signal for
learning to select features comes from the comparison of the answers given by
the predictor and the ground-truth answers. In this work, we propose to squeeze
more information from the predictor via an information calibration method. More
precisely, we train two models jointly: one is a typical neural model that
solves the task at hand in an accurate but black-box manner, and the other is a
selector-predictor model that additionally produces a rationale for its
prediction. The first model is used as a guide to the second model. We use an
adversarial-based technique to calibrate the information extracted by the two
models such that the difference between them is an indicator of the missed or
over-selected features. In addition, for natural language tasks, we propose to
use a language-model-based regularizer to encourage the extraction of fluent
rationales. Experimental results on a sentiment analysis task as well as on
three tasks from the legal domain show the effectiveness of our approach to
rationale extraction.

negative:
Hierarchical neural architectures are often used to capture long-distance
dependencies and have been applied to many document-level tasks such as
summarization, document segmentation, and sentiment analysis. However,
effective usage of such a large context can be difficult to learn, especially
in the case where there is limited labeled data available. Building on the
recent success of language model pretraining methods for learning flat
representations of text, we propose algorithms for pre-training hierarchical
document representations from unlabeled data. Unlike prior work, which has
focused on pre-training contextual token representations or context-independent
{sentence/paragraph} representations, our hierarchical document representations
include fixed-length sentence/paragraph representations which integrate
contextual information from the entire documents. Experiments on document
segmentation, document-level question answering, and extractive document
summarization demonstrate the effectiveness of the proposed pre-training
algorithms.

negative:
Modern machine learning algorithms are capable of providing remarkably
accurate point-predictions; however, questions remain about their statistical
reliability. Unlike conventional machine learning methods, conformal prediction
algorithms return confidence sets (i.e., set-valued predictions) that
correspond to a given significance level. Moreover, these confidence sets are
valid in the sense that they guarantee finite sample control over type 1 error
probabilities, allowing the practitioner to choose an acceptable error rate. In
our paper, we propose inductive conformal prediction (ICP) algorithms for the
tasks of text infilling and part-of-speech (POS) prediction for natural
language data. We construct new conformal prediction-enhanced bidirectional
encoder representations from transformers (BERT) and bidirectional long
short-term memory (BiLSTM) algorithms for POS tagging and a new conformal
prediction-enhanced BERT algorithm for text infilling. We analyze the
performance of the algorithms in simulations using the Brown Corpus, which
contains over 57,000 sentences. Our results demonstrate that the ICP algorithms
are able to produce valid set-valued predictions that are small enough to be
applicable in real-world applications. We also provide a real data example for
how our proposed set-valued predictions can improve machine generated audio
transcriptions.

negative:
In this paper, we present a novel approach for conformal prediction (CP), in
which we aim to identify a set of promising prediction candidates -- in place
of a single prediction. This set is guaranteed to contain a correct answer with
high probability, and is well-suited for many open-ended classification tasks.
In the standard CP paradigm, the predicted set can often be unusably large and
also costly to obtain. This is particularly pervasive in settings where the
correct answer is not unique, and the number of total possible answers is high.
We first expand the CP correctness criterion to allow for additional, inferred
"admissible" answers, which can substantially reduce the size of the predicted
set while still providing valid performance guarantees. Second, we amortize
costs by conformalizing prediction cascades, in which we aggressively prune
implausible labels early on by using progressively stronger classifiers --
again, while still providing valid performance guarantees. We demonstrate the
empirical effectiveness of our approach for multiple applications in natural
language processing and computational chemistry for drug discovery.

negative:
The great success of Convolutional Neural Networks (CNN) for facial attribute
prediction relies on a large amount of labeled images. Facial image datasets
are usually annotated by some commonly used attributes (e.g., gender), while
labels for the other attributes (e.g., big nose) are limited which causes their
prediction challenging. To address this problem, we use a new Multi-Task
Learning (MTL) paradigm in which a facial attribute predictor uses the
knowledge of other related attributes to obtain a better generalization
performance. Here, we leverage MLT paradigm in two problem settings. First, it
is assumed that the structure of the tasks (e.g., grouping pattern of facial
attributes) is known as a prior knowledge, and parameters of the tasks (i.e.,
predictors) within the same group are represented by a linear combination of a
limited number of underlying basis tasks. Here, a sparsity constraint on the
coefficients of this linear combination is also considered such that each task
is represented in a more structured and simpler manner. Second, it is assumed
that the structure of the tasks is unknown, and then structure and parameters
of the tasks are learned jointly by using a Laplacian regularization framework.
Our MTL methods are compared with competing methods for facial attribute
prediction to show its effectiveness.

negative:
The goal of our research is to develop methods advancing automatic visual
recognition. In order to predict the unique or multiple labels associated to an
image, we study different kind of Deep Neural Networks architectures and
methods for supervised features learning. We first draw up a state-of-the-art
review of the Convolutional Neural Networks aiming to understand the history
behind this family of statistical models, the limit of modern architectures and
the novel techniques currently used to train deep CNNs. The originality of our
work lies in our approach focusing on tasks with a low amount of data. We
introduce different models and techniques to achieve the best accuracy on
several kind of datasets, such as a medium dataset of food recipes (100k
images) for building a web API, or a small dataset of satellite images (6,000)
for the DSG online challenge that we've won. We also draw up the
state-of-the-art in Weakly Supervised Learning, introducing different kind of
CNNs able to localize regions of interest. Our last contribution is a
framework, build on top of Torch7, for training and testing deep models on any
visual recognition tasks and on datasets of any scale.

negative:
Recurrent Neural Networks were, until recently, one of the best ways to
capture the timely dependencies in sequences. However, with the introduction of
the Transformer, it has been proven that an architecture with only
attention-mechanisms without any RNN can improve on the results in various
sequence processing tasks (e.g. NLP). Multiple studies since then have shown
that similar approaches can be applied for images, point clouds, video, audio
or time series forecasting. Furthermore, solutions such as the Perceiver or the
Informer have been introduced to expand on the applicability of the
Transformer. Our main objective is testing and evaluating the effectiveness of
applying Transformer-like models on time series data, tackling susceptibility
to anomalies, context awareness and space complexity by fine-tuning the
hyperparameters, preprocessing the data, applying dimensionality reduction or
convolutional encodings, etc. We are also looking at the problem of next-frame
prediction and exploring ways to modify existing solutions in order to achieve
higher performance and learn generalized knowledge.

negative:
Transformer, which originates from machine translation, is particularly
powerful at modeling long-range dependencies. Currently, the transformer is
making revolutionary progress in various vision tasks, leading to significant
performance improvements compared with the convolutional neural network (CNN)
based frameworks. In this paper, we conduct extensive research on exploiting
the contributions of transformers for accurate and reliable salient object
detection. For the former, we apply transformer to a deterministic model, and
explain that the effective structure modeling and global context modeling
abilities lead to its superior performance compared with the CNN based
frameworks. For the latter, we observe that both CNN and transformer based
frameworks suffer greatly from the over-confidence issue, where the models tend
to generate wrong predictions with high confidence. To estimate the reliability
degree of both CNN- and transformer-based frameworks, we further present a
latent variable model, namely inferential generative adversarial network
(iGAN), based on the generative adversarial network (GAN). The stochastic
attribute of the latent variable makes it convenient to estimate the predictive
uncertainty, serving as an auxiliary output to evaluate the reliability of
model prediction. Different from the conventional GAN, which defines the
distribution of the latent variable as fixed standard normal distribution
$\mathcal{N}(0,\mathbf{I})$, the proposed iGAN infers the latent variable by
gradient-based Markov Chain Monte Carlo (MCMC), namely Langevin dynamics,
leading to an input-dependent latent variable model. We apply our proposed iGAN
to both fully and weakly supervised salient object detection, and explain that
iGAN within the transformer framework leads to both accurate and reliable
salient object detection.

negative:
In medical applications, deep learning methods are built to automate
diagnostic tasks. However, a clinically relevant question that practitioners
usually face, is how to predict the future trajectory of a disease (prognosis).
Current methods for such a problem often require domain knowledge, and are
complicated to apply. In this paper, we formulate the prognosis prediction
problem as a one-to-many forecasting problem from multimodal data. Inspired by
a clinical decision-making process with two agents -- a radiologist and a
general practitioner, we model a prognosis prediction problem with two
transformer-based components that share information between each other. The
first block in this model aims to analyze the imaging data, and the second
block leverages the internal representations of the first one as inputs, also
fusing them with auxiliary patient data. We show the effectiveness of our
method in predicting the development of structural knee osteoarthritis changes
over time. Our results show that the proposed method outperforms the
state-of-the-art baselines in terms of various performance metrics. In
addition, we empirically show that the existence of the multi-agent
transformers with depths of 2 is sufficient to achieve good performances. Our
code is publicly available at \url{https://github.com/MIPT-Oulu/CLIMAT}.

negative:
Deep Neural Networks achieve state-of-the-art results in many different
problem settings by exploiting vast amounts of training data. However,
collecting, storing and - in the case of supervised learning - labelling the
data is expensive and time-consuming. Additionally, assessing the networks'
generalization abilities or predicting how the inferred output changes under
input transformations is complicated since the networks are usually treated as
a black box. Both of these problems can be mitigated by incorporating prior
knowledge into the neural network. One promising approach, inspired by the
success of convolutional neural networks in computer vision tasks, is to
incorporate knowledge about symmetric geometrical transformations of the
problem to solve that affect the output in a predictable way. This promises an
increased data efficiency and more interpretable network outputs. In this
survey, we try to give a concise overview about different approaches that
incorporate geometrical prior knowledge into neural networks. Additionally, we
connect those methods to 3D object detection for autonomous driving, where we
expect promising results when applying those methods.

negative:
The data distribution commonly evolves over time leading to problems such as
concept drift that often decrease classifier performance. Current techniques
are not adequate for this problem because they either require detailed
knowledge of the transformation or are not suited for anticipating unseen
domains but can only adapt to domains, where data samples are available. We
seek to predict unseen data (and their labels) allowing us to tackle challenges
s a non-constant data distribution in a proactive manner rather than detecting
and reacting to already existing changes that might already have led to errors.
To this end, we learn a domain transformer in an unsupervised manner that
allows generating data of unseen domains. Our approach first matches
independently learned latent representations of two given domains obtained from
an auto-encoder using a Cycle-GAN. In turn, a transformation of the original
samples can be learned that can be applied iteratively to extrapolate to unseen
domains. Our evaluation of CNNs on image data confirms the usefulness of the
approach. It also achieves very good results on the well-known problem of
unsupervised domain adaption, where only labels but no samples have to be
predicted. Code is available at https://github.com/JohnTailor/DoTra.

negative:
This paper presents a new state-of-the-art for document image classification
and retrieval, using features learned by deep convolutional neural networks
(CNNs). In object and scene analysis, deep neural nets are capable of learning
a hierarchical chain of abstraction from pixel inputs to concise and
descriptive representations. The current work explores this capacity in the
realm of document analysis, and confirms that this representation strategy is
superior to a variety of popular hand-crafted alternatives. Experiments also
show that (i) features extracted from CNNs are robust to compression, (ii) CNNs
trained on non-document images transfer well to document analysis tasks, and
(iii) enforcing region-specific feature-learning is unnecessary given
sufficient training data. This work also makes available a new labelled subset
of the IIT-CDIP collection, containing 400,000 document images across 16
categories, useful for training new CNNs for document analysis.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
Deep reading models for question-answering have demonstrated promising
performance over the last couple of years. However current systems tend to
learn how to cleverly extract a span of the source document, based on its
similarity with the question, instead of seeking for the appropriate answer.
Indeed, a reading machine should be able to detect relevant passages in a
document regarding a question, but more importantly, it should be able to
reason over the important pieces of the document in order to produce an answer
when it is required. To motivate this purpose, we present ReviewQA, a
question-answering dataset based on hotel reviews. The questions of this
dataset are linked to a set of relational understanding competencies that we
expect a model to master. Indeed, each question comes with an associated type
that characterizes the required competency. With this framework, it is possible
to benchmark the main families of models and to get an overview of what are the
strengths and the weaknesses of a given model on the set of tasks evaluated in
this dataset. Our corpus contains more than 500.000 questions in natural
language over 100.000 hotel reviews. Our setup is projective, the answer of a
question does not need to be extracted from a document, like in most of the
recent datasets, but selected among a set of candidates that contains all the
possible answers to the questions of the dataset. Finally, we present several
baselines over this dataset.

negative:
Under special circumstances, summaries should conform to a particular style
with patterns, such as court judgments and abstracts in academic papers. To
this end, the prototype document-summary pairs can be utilized to generate
better summaries. There are two main challenges in this task: (1) the model
needs to incorporate learned patterns from the prototype, but (2) should avoid
copying contents other than the patternized words---such as irrelevant
facts---into the generated summaries. To tackle these challenges, we design a
model named Prototype Editing based Summary Generator (PESG). PESG first learns
summary patterns and prototype facts by analyzing the correlation between a
prototype document and its summary. Prototype facts are then utilized to help
extract facts from the input document. Next, an editing generator generates new
summary based on the summary pattern or extracted facts. Finally, to address
the second challenge, a fact checker is used to estimate mutual information
between the input document and generated summary, providing an additional
signal for the generator. Extensive experiments conducted on a large-scale
real-world text summarization dataset show that PESG achieves the
state-of-the-art performance in terms of both automatic metrics and human
evaluations.

negative:
Conformal prediction is a method of producing prediction sets that can be
applied on top of a wide range of prediction algorithms. The method has a
guaranteed coverage probability under the standard IID assumption regardless of
whether the assumptions (often considerably more restrictive) of the underlying
algorithm are satisfied. However, for the method to be really useful it is
desirable that in the case where the assumptions of the underlying algorithm
are satisfied, the conformal predictor loses little in efficiency as compared
with the underlying algorithm (whereas being a conformal predictor, it has the
stronger guarantee of validity). In this paper we explore the degree to which
this additional requirement of efficiency is satisfied in the case of Bayesian
ridge regression; we find that asymptotically conformal prediction sets differ
little from ridge regression prediction intervals when the standard Bayesian
assumptions are satisfied.

negative:
In the context of investigative journalism, we address the problem of
automatically identifying which claims in a given document are most worthy and
should be prioritized for fact-checking. Despite its importance, this is a
relatively understudied problem. Thus, we create a new dataset of political
debates, containing statements that have been fact-checked by nine reputable
sources, and we train machine learning models to predict which claims should be
prioritized for fact-checking, i.e., we model the problem as a ranking task.
Unlike previous work, which has looked primarily at sentences in isolation, in
this paper we focus on a rich input representation modeling the context:
relationship between the target statement and the larger context of the debate,
interaction between the opponents, and reaction by the moderator and by the
public. Our experiments show state-of-the-art results, outperforming a strong
rivaling system by a margin, while also confirming the importance of the
contextual information.

negative:
Text classification is an important and classical problem in natural language
processing. There have been a number of studies that applied convolutional
neural networks (convolution on regular grid, e.g., sequence) to
classification. However, only a limited number of studies have explored the
more flexible graph convolutional neural networks (convolution on non-grid,
e.g., arbitrary graph) for the task. In this work, we propose to use graph
convolutional networks for text classification. We build a single text graph
for a corpus based on word co-occurrence and document word relations, then
learn a Text Graph Convolutional Network (Text GCN) for the corpus. Our Text
GCN is initialized with one-hot representation for word and document, it then
jointly learns the embeddings for both words and documents, as supervised by
the known class labels for documents. Our experimental results on multiple
benchmark datasets demonstrate that a vanilla Text GCN without any external
word embeddings or knowledge outperforms state-of-the-art methods for text
classification. On the other hand, Text GCN also learns predictive word and
document embeddings. In addition, experimental results show that the
improvement of Text GCN over state-of-the-art comparison methods become more
prominent as we lower the percentage of training data, suggesting the
robustness of Text GCN to less training data in text classification.

negative:
We explore using multilingual document embeddings for nearest neighbor mining
of parallel data. Three document-level representations are investigated: (i)
document embeddings generated by simply averaging multilingual sentence
embeddings; (ii) a neural bag-of-words (BoW) document encoding model; (iii) a
hierarchical multilingual document encoder (HiDE) that builds on our
sentence-level model. The results show document embeddings derived from
sentence-level averaging are surprisingly effective for clean datasets, but
suggest models trained hierarchically at the document-level are more effective
on noisy data. Analysis experiments demonstrate our hierarchical models are
very robust to variations in the underlying sentence embedding quality. Using
document embeddings trained with HiDE achieves state-of-the-art performance on
United Nations (UN) parallel document mining, 94.9% P@1 for en-fr and 97.3% P@1
for en-es.

negative:
With the increase of information, document classification as one of the
methods of text mining, plays vital role in many management and organizing
information. Document classification is the process of assigning a document to
one or more predefined category labels. Document classification includes
different parts such as text processing, term selection, term weighting and
final classification. The accuracy of document classification is very
important. Thus improvement in each part of classification should lead to
better results and higher precision. Term weighting has a great impact on the
accuracy of the classification. Most of the existing weighting methods exploit
the statistical information of terms in documents and do not consider semantic
relations between words. In this paper, an automated document classification
system is presented that uses a novel term weighting method based on semantic
relations between terms. To evaluate the proposed method, three standard
Persian corpuses are used. Experiment results show 2 to 4 percent improvement
in classification accuracy compared with the best previous designed system for
Persian documents.

negative:
The property of conformal predictors to guarantee the required accuracy rate
makes this framework attractive in various practical applications. However,
this property is achieved at a price of reduction in precision. In the case of
conformal classification, the systems can output multiple class labels instead
of one. It is also known from the literature, that the choice of nonconformity
function has a major impact on the efficiency of conformal classifiers.
Recently, it was shown that different model-agnostic nonconformity functions
result in conformal classifiers with different characteristics. For a Neural
Network-based conformal classifier, the inverse probability (or hinge loss)
allows minimizing the average number of predicted labels, and margin results in
a larger fraction of singleton predictions. In this work, we aim to further
extend this study. We perform an experimental evaluation using 8 different
classification algorithms and discuss when the previously observed relationship
holds or not. Additionally, we propose a successful method to combine the
properties of these two nonconformity functions. The experimental evaluation is
done using 11 real and 5 synthetic datasets.

negative:
This paper summarizes our entries to both subtasks of the first DialDoc
shared task which focuses on the agent response prediction task in
goal-oriented document-grounded dialogs. The task is split into two subtasks:
predicting a span in a document that grounds an agent turn and generating an
agent response based on a dialog and grounding document. In the first subtask,
we restrict the set of valid spans to the ones defined in the dataset, use a
biaffine classifier to model spans, and finally use an ensemble of different
models. For the second subtask, we use a cascaded model which grounds the
response prediction on the predicted span instead of the full document. With
these approaches, we obtain significant improvements in both subtasks compared
to the baseline.

negative:
In the Query Focused Multi-Document Summarization (QF-MDS) task, a set of
documents and a query are given where the goal is to generate a summary from
these documents based on the given query. However, one major challenge for this
task is the lack of availability of labeled training datasets. To overcome this
issue, in this paper, we propose a novel weakly supervised learning approach
via utilizing distant supervision. In particular, we use datasets similar to
the target dataset as the training data where we leverage pre-trained sentence
similarity models to generate the weak reference summary of each individual
document in a document set from the multi-document gold reference summaries.
Then, we iteratively train our summarization model on each single-document to
alleviate the computational complexity issue that occurs while training neural
summarization models in multiple documents (i.e., long sequences) at once.
Experimental results in Document Understanding Conferences (DUC) datasets show
that our proposed approach sets a new state-of-the-art result in terms of
various evaluation metrics.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
Most existing examples of full conformal predictive systems, split-conformal
predictive systems, and cross-conformal predictive systems impose severe
restrictions on the adaptation of predictive distributions to the test object
at hand. In this paper we develop split-conformal and cross-conformal
predictive systems that are fully adaptive. Our method consists in calibrating
existing predictive systems; the input predictive system is not supposed to
satisfy any properties of validity, whereas the output predictive system is
guaranteed to be calibrated in probability. It is interesting that the method
may also work without the IID assumption, standard in conformal prediction.

negative:
Neural conversation models are known to generate appropriate but
non-informative responses in general. A scenario where informativeness can be
significantly enhanced is Conversing by Reading (CbR), where conversations take
place with respect to a given external document. In previous work, the external
document is utilized by (1) creating a context-aware document memory that
integrates information from the document and the conversational context, and
then (2) generating responses referring to the memory. In this paper, we
propose to create the document memory with some anticipated responses in mind.
This is achieved using a teacher-student framework. The teacher is given the
external document, the context, and the ground-truth response, and learns how
to build a response-aware document memory from three sources of information.
The student learns to construct a response-anticipated document memory from the
first two sources, and the teacher's insight on memory creation. Empirical
results show that our model outperforms the previous state-of-the-art for the
CbR task.

negative:
Conformal predictors are set predictors that are automatically valid in the
sense of having coverage probability equal to or exceeding a given confidence
level. Inductive conformal predictors are a computationally efficient version
of conformal predictors satisfying the same property of validity. However,
inductive conformal predictors have been only known to control unconditional
coverage probability. This paper explores various versions of conditional
validity and various ways to achieve them using inductive conformal predictors
and their modifications.

negative:
When one observes a sequence of variables $(x_1, y_1), \ldots, (x_n, y_n)$,
Conformal Prediction (CP) is a methodology that allows to estimate a confidence
set for $y_{n+1}$ given $x_{n+1}$ by merely assuming that the distribution of
the data is exchangeable. CP sets have guaranteed coverage for any finite
population size $n$. While appealing, the computation of such a set turns out
to be infeasible in general, e.g. when the unknown variable $y_{n+1}$ is
continuous. The bottleneck is that it is based on a procedure that readjusts a
prediction model on data where we replace the unknown target by all its
possible values in order to select the most probable one. This requires
computing an infinite number of models, which often makes it intractable. In
this paper, we combine CP techniques with classical algorithmic stability
bounds to derive a prediction set computable with a single model fit. We
demonstrate that our proposed confidence set does not lose any coverage
guarantees while avoiding the need for data splitting as currently done in the
literature. We provide some numerical experiments to illustrate the tightness
of our estimation when the sample size is sufficiently large, on both synthetic
and real datasets.

negative:
An important challenge facing modern machine learning is how to rigorously
quantify the uncertainty of model predictions. Conveying uncertainty is
especially important when there are changes to the underlying data distribution
that might invalidate the predictive model. Yet, most existing uncertainty
quantification algorithms break down in the presence of such shifts. We propose
a novel approach that addresses this challenge by constructing \emph{probably
approximately correct (PAC)} prediction sets in the presence of covariate
shift. Our approach focuses on the setting where there is a covariate shift
from the source distribution (where we have labeled training examples) to the
target distribution (for which we want to quantify uncertainty). Our algorithm
assumes given importance weights that encode how the probabilities of the
training examples change under the covariate shift. In practice, importance
weights typically need to be estimated; thus, we extend our algorithm to the
setting where we are given confidence intervals for the importance weights. We
demonstrate the effectiveness of our approach on covariate shifts based on
DomainNet and ImageNet. Our algorithm satisfies the PAC constraint, and gives
prediction sets with the smallest average normalized size among approaches that
always satisfy the PAC constraint.

negative:
Convolutional neural networks (CNN) are the dominant deep neural network
(DNN) architecture for computer vision. Recently, Transformer and multi-layer
perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer,
started to lead new trends as they showed promising results in the ImageNet
classification task. In this paper, we conduct empirical studies on these DNN
structures and try to understand their respective pros and cons. To ensure a
fair comparison, we first develop a unified framework called SPACH which adopts
separate modules for spatial and channel processing. Our experiments under the
SPACH framework reveal that all structures can achieve competitive performance
at a moderate scale. However, they demonstrate distinctive behaviors when the
network size scales up. Based on our findings, we propose two hybrid models
using convolution and Transformer modules. The resulting Hybrid-MS-S+ model
achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is
already on par with the SOTA models with sophisticated designs. The code and
models are publicly available at https://github.com/microsoft/SPACH.

negative:
Document categorization, which aims to assign a topic label to each document,
plays a fundamental role in a wide variety of applications. Despite the success
of existing studies in conventional supervised document classification, they
are less concerned with two real problems: (1) the presence of metadata: in
many domains, text is accompanied by various additional information such as
authors and tags. Such metadata serve as compelling topic indicators and should
be leveraged into the categorization framework; (2) label scarcity: labeled
training samples are expensive to obtain in some cases, where categorization
needs to be performed using only a small set of annotated data. In recognition
of these two challenges, we propose MetaCat, a minimally supervised framework
to categorize text with metadata. Specifically, we develop a generative process
describing the relationships between words, documents, labels, and metadata.
Guided by the generative model, we embed text and metadata into the same
semantic space to encode heterogeneous signals. Then, based on the same
generative process, we synthesize training samples to address the bottleneck of
label scarcity. We conduct a thorough evaluation on a wide range of datasets.
Experimental results prove the effectiveness of MetaCat over many competitive
baselines.

negative:
Data representation is a fundamental task in machine learning. The
representation of data affects the performance of the whole machine learning
system. In a long history, the representation of data is done by feature
engineering, and researchers aim at designing better features for specific
tasks. Recently, the rapid development of deep learning and representation
learning has brought new inspiration to various domains.
  In natural language processing, the most widely used feature representation
is the Bag-of-Words model. This model has the data sparsity problem and cannot
keep the word order information. Other features such as part-of-speech tagging
or more complex syntax features can only fit for specific tasks in most cases.
This thesis focuses on word representation and document representation. We
compare the existing systems and present our new model.
  First, for generating word embeddings, we make comprehensive comparisons
among existing word embedding models. In terms of theory, we figure out the
relationship between the two most important models, i.e., Skip-gram and GloVe.
In our experiments, we analyze three key points in generating word embeddings,
including the model construction, the training corpus and parameter design. We
evaluate word embeddings with three types of tasks, and we argue that they
cover the existing use of word embeddings. Through theory and practical
experiments, we present some guidelines for how to generate a good word
embedding.
  Second, in Chinese character or word representation. We introduce the joint
training of Chinese character and word. ...
  Third, for document representation, we analyze the existing document
representation models, including recursive NNs, recurrent NNs and convolutional
NNs. We point out the drawbacks of these models and present our new model, the
recurrent convolutional neural networks. ...

negative:
One of the major prerequisites for any deep learning approach is the
availability of large-scale training data. When dealing with scanned document
images in real world scenarios, the principal information of its content is
stored in the layout itself. In this work, we have proposed an automated deep
generative model using Graph Neural Networks (GNNs) to generate synthetic data
with highly variable and plausible document layouts that can be used to train
document interpretation systems, in this case, specially in digital mailroom
applications. It is also the first graph-based approach for document layout
generation task experimented on administrative document images, in this case,
invoices.

negative:
Predicting the quality of a text document is a critical task when presented
with the problem of measuring the performance of a document before its release.
In this work, we evaluate various features including those extracted from the
text content (textual) and those describing higher-level characteristics of the
text (meta) features that are not directly available from the text, and show
how these features inform prediction of document quality in different ways.
Moreover, we also compare our methods on both social user-generated data such
as tweets, and scholarly user-generated data such as academic articles, showing
how the same features differently influence prediction of quality across these
disparate domains.

negative:
This paper applies conformal prediction techniques to compute simultaneous
prediction bands and clustering trees for functional data. These tools can be
used to detect outliers and clusters. Both our prediction bands and clustering
trees provide prediction sets for the underlying stochastic process with a
guaranteed finite sample behavior, under no distributional assumptions. The
prediction sets are also informative in that they correspond to the high
density region of the underlying process. While ordinary conformal prediction
has high computational cost for functional data, we use the inductive conformal
predictor, together with several novel choices of conformity scores, to
simplify the computation. Our methods are illustrated on some real data
examples.

negative:
Document grounded generation is the task of using the information provided in
a document to improve text generation. This work focuses on two different
document grounded generation tasks: Wikipedia Update Generation task and
Dialogue response generation. Our work introduces two novel adaptations of
large scale pre-trained encoder-decoder models focusing on building context
driven representation of the document and enabling specific attention to the
information in the document. Additionally, we provide a stronger BART baseline
for these tasks. Our proposed techniques outperform existing methods on both
automated (at least 48% increase in BLEU-4 points) and human evaluation for
closeness to reference and relevance to the document. Furthermore, we perform
comprehensive manual inspection of the generated output and categorize errors
to provide insights into future directions in modeling these tasks.

negative:
In real-world settings involving consequential decision-making, the
deployment of machine learning systems generally requires both reliable
uncertainty quantification and protection of individuals' privacy. We present a
framework that treats these two desiderata jointly. Our framework is based on
conformal prediction, a methodology that augments predictive models to return
prediction sets that provide uncertainty quantification -- they provably cover
the true response with a user-specified probability, such as 90%. One might
hope that when used with privately-trained models, conformal prediction would
yield privacy guarantees for the resulting prediction sets; unfortunately this
is not the case. To remedy this key problem, we develop a method that takes any
pre-trained predictive model and outputs differentially private prediction
sets. Our method follows the general approach of split conformal prediction; we
use holdout data to calibrate the size of the prediction sets but preserve
privacy by using a privatized quantile subroutine. This subroutine compensates
for the noise introduced to preserve privacy in order to guarantee correct
coverage. We evaluate the method on large-scale computer vision datasets.

negative:
When tracking user-specific online activities, each user's preference is
revealed in the form of choices and comparisons. For example, a user's purchase
history is a record of her choices, i.e. which item was chosen among a subset
of offerings. A user's preferences can be observed either explicitly as in
movie ratings or implicitly as in viewing times of news articles. Given such
individualized ordinal data in the form of comparisons and choices, we address
the problem of collaboratively learning representations of the users and the
items. The learned features can be used to predict a user's preference of an
unseen item to be used in recommendation systems. This also allows one to
compute similarities among users and items to be used for categorization and
search. Motivated by the empirical successes of the MultiNomial Logit (MNL)
model in marketing and transportation, and also more recent successes in word
embedding and crowdsourced image embedding, we pose this problem as learning
the MNL model parameters that best explain the data. We propose a convex
relaxation for learning the MNL model, and show that it is minimax optimal up
to a logarithmic factor by comparing its performance to a fundamental lower
bound. This characterizes the minimax sample complexity of the problem, and
proves that the proposed estimator cannot be improved upon other than by a
logarithmic factor. Further, the analysis identifies how the accuracy depends
on the topology of sampling via the spectrum of the sampling graph. This
provides a guideline for designing surveys when one can choose which items are
to be compared. This is accompanied by numerical simulations on synthetic and
real data sets, confirming our theoretical predictions.

negative:
Recent progress in pretrained Transformer-based language models has shown
great success in learning contextual representation of text. However, due to
the quadratic self-attention complexity, most of the pretrained Transformers
models can only handle relatively short text. It is still a challenge when it
comes to modeling very long documents. In this work, we propose to use a graph
attention network on top of the available pretrained Transformers model to
learn document embeddings. This graph attention network allows us to leverage
the high-level semantic structure of the document. In addition, based on our
graph document model, we design a simple contrastive learning strategy to
pretrain our models on a large amount of unlabeled corpus. Empirically, we
demonstrate the effectiveness of our approaches in document classification and
document retrieval tasks.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.

negative:
Recent advances in attention-based networks have shown that Vision
Transformers can achieve state-of-the-art or near state-of-the-art results on
many image classification tasks. This puts transformers in the unique position
of being a promising alternative to traditional convolutional neural networks
(CNNs). While CNNs have been carefully studied with respect to adversarial
attacks, the same cannot be said of Vision Transformers. In this paper, we
study the robustness of Vision Transformers to adversarial examples. Our
analyses of transformer security is divided into three parts. First, we test
the transformer under standard white-box and black-box attacks. Second, we
study the transferability of adversarial examples between CNNs and
transformers. We show that adversarial examples do not readily transfer between
CNNs and transformers. Based on this finding, we analyze the security of a
simple ensemble defense of CNNs and transformers. By creating a new attack, the
self-attention blended gradient attack, we show that such an ensemble is not
secure under a white-box adversary. However, under a black-box adversary, we
show that an ensemble can achieve unprecedented robustness without sacrificing
clean accuracy. Our analysis for this work is done using six types of white-box
attacks and two types of black-box attacks. Our study encompasses multiple
Vision Transformers, Big Transfer Models and CNN architectures trained on
CIFAR-10, CIFAR-100 and ImageNet.

negative:
We develop a novel approach for confidently accelerating inference in the
large and expensive multilayer Transformers that are now ubiquitous in natural
language processing (NLP). Amortized or approximate computational methods
increase efficiency, but can come with unpredictable performance costs. In this
work, we present CATs -- Confident Adaptive Transformers -- in which we
simultaneously increase computational efficiency, while guaranteeing a
specifiable degree of consistency with the original model with high confidence.
Our method trains additional prediction heads on top of intermediate layers,
and dynamically decides when to stop allocating computational effort to each
input using a meta consistency classifier. To calibrate our early prediction
stopping rule, we formulate a unique extension of conformal prediction. We
demonstrate the effectiveness of this approach on four classification and
regression tasks.

negative:
Quantifying uncertainty in model predictions is a common goal for
practitioners seeking more than just point predictions. One tool for
uncertainty quantification that requires minimal assumptions is conformal
inference, which can help create probabilistically valid prediction regions for
black box models. Classical conformal prediction only provides marginal
validity, whereas in many situations locally valid prediction regions are
desirable. Deciding how best to partition the feature space X when applying
localized conformal prediction is still an open question. We present MD-split+,
a practical local conformal approach that creates X partitions based on
localized model performance of conditional density estimation models. Our
method handles complex real-world data settings where such models may be
misspecified, and scales to high-dimensional inputs. We discuss how our local
partitions philosophically align with expected behavior from an unattainable
conditional conformal inference approach. We also empirically compare our
method against other local conformal approaches.

negative:
Relating entities and events in text is a key component of natural language
understanding. Cross-document coreference resolution, in particular, is
important for the growing interest in multi-document analysis tasks. In this
work we propose a new model that extends the efficient sequential prediction
paradigm for coreference resolution to cross-document settings and achieves
competitive results for both entity and event coreference while provides strong
evidence of the efficacy of both sequential models and higher-order inference
in cross-document settings. Our model incrementally composes mentions into
cluster representations and predicts links between a mention and the already
constructed clusters, approximating a higher-order model. In addition, we
conduct extensive ablation studies that provide new insights into the
importance of various inputs and representation types in coreference.

negative:
Syntactic annotation of corpora in the form of part-of-speech (POS) tags is a
key requirement for both linguistic research and subsequent automated natural
language processing (NLP) tasks. This problem is commonly tackled using machine
learning methods, i.e., by training a POS tagger on a sufficiently large corpus
of labeled data. While the problem of POS tagging can essentially be considered
as solved for modern languages, historical corpora turn out to be much more
difficult, especially due to the lack of native speakers and sparsity of
training data. Moreover, most texts have no sentences as we know them today,
nor a common orthography. These irregularities render the task of automated POS
tagging more difficult and error-prone. Under these circumstances, instead of
forcing the POS tagger to predict and commit to a single tag, it should be
enabled to express its uncertainty. In this paper, we consider POS tagging
within the framework of set-valued prediction, which allows the POS tagger to
express its uncertainty via predicting a set of candidate POS tags instead of
guessing a single one. The goal is to guarantee a high confidence that the
correct POS tag is included while keeping the number of candidates small. In
our experimental study, we find that extending state-of-the-art POS taggers to
set-valued prediction yields more precise and robust taggings, especially for
unknown words, i.e., words not occurring in the training data.

negative:
We study distribution free, nonparametric prediction bands with a special
focus on their finite sample behavior. First we investigate and develop
different notions of finite sample coverage guarantees. Then we give a new
prediction band estimator by combining the idea of "conformal prediction" (Vovk
et al. 2009) with nonparametric conditional density estimation. The proposed
estimator, called COPS (Conformal Optimized Prediction Set), always has finite
sample guarantee in a stronger sense than the original conformal prediction
estimator. Under regularity conditions the estimator converges to an oracle
band at a minimax optimal rate. A fast approximation algorithm and a data
driven method for selecting the bandwidth are developed. The method is
illustrated first in simulated data. Then, an application shows that the
proposed method gives desirable prediction intervals in an automatic way, as
compared to the classical linear regression modeling.

negative:
In recent developments in the field of Computer Vision, a rise is seen in the
use of transformer-based architectures. They are surpassing the
state-of-the-art set by CNN architectures in accuracy but on the other hand,
they are computationally very expensive to train from scratch. As these models
are quite recent in the Computer Vision field, there is a need to study it's
transfer learning capabilities and compare it with CNNs so that we can
understand which architecture is better when applied to real world problems
with small data. In this work, we follow a simple yet restrictive method for
fine-tuning both CNN and Transformer models pretrained on ImageNet1K on
CIFAR-10 and compare them with each other. We only unfreeze the last
transformer/encoder or last convolutional block of a model and freeze all the
layers before it while adding a simple MLP at the end for classification. This
simple modification lets us use the raw learned weights of both these neural
networks. From our experiments, we find out that transformers-based
architectures not only achieve higher accuracy than CNNs but some transformers
even achieve this feat with around 4 times lesser number of parameters.

negative:
The increasing amount of online content motivated the development of
multi-document summarization methods. In this work, we explore straightforward
approaches to extend single-document summarization methods to multi-document
summarization. The proposed methods are based on the hierarchical combination
of single-document summaries, and achieves state of the art results.

negative:
Measuring similarity between texts is an important task for several
applications. Available approaches to measure document similarity are
inadequate for document pairs that have non-comparable lengths, such as a long
document and its summary. This is because of the lexical, contextual and the
abstraction gaps between a long document of rich details and its concise
summary of abstract information. In this paper, we present a document matching
approach to bridge this gap, by comparing the texts in a common space of hidden
topics. We evaluate the matching algorithm on two matching tasks and find that
it consistently and widely outperforms strong baselines. We also highlight the
benefits of incorporating domain knowledge to text matching.

negative:
The vast volume of data generated by modern astronomical surveys offers test
beds for the application of machine-learning. It is important to evaluate
potential existing tools and determine those that are optimal for extracting
scientific knowledge from the available observations. We explore the
possibility of using clustering algorithms to separate stellar populations with
distinct chemical patterns. Star clusters are likely the most chemically
homogeneous populations in the Galaxy, and therefore any practical approach to
identifying distinct stellar populations should at least be able to separate
clusters from each other. We applied eight clustering algorithms combined with
four dimensionality reduction strategies to automatically distinguish stellar
clusters using chemical abundances of 13 elements. Our sample includes 18
stellar clusters with a total of 453 stars. We use statistical tests showing
that some pairs of clusters are indistinguishable from each other when chemical
abundances from the Apache Point Galactic Evolution Experiment (APOGEE) are
used. However, for most clusters we are able to automatically assign membership
with metric scores similar to previous works. The confusion level of the
automatically selected clusters is consistent with statistical tests that
demonstrate the impossibility of perfectly distinguishing all the clusters from
each other. These statistical tests and confusion levels establish a limit for
the prospect of blindly identifying stars born in the same cluster based solely
on chemical abundances. We find that some of the algorithms we explored are
capable of blindly identify stellar populations with similar ages and chemical
distributions in the APOGEE data. Because some stellar clusters are chemically
indistinguishable, our study supports the notion of extending weak chemical
tagging that involves families of clusters instead of individual clusters

negative:
We present a contrasting learning approach with data augmentation techniques
to learn document representations in an unsupervised manner. Inspired by recent
contrastive self-supervised learning algorithms used for image and NLP
pretraining, we hypothesize that high-quality document embedding should be
invariant to diverse paraphrases that preserve the semantics of the original
document. With different backbones and contrastive learning frameworks, our
study reveals the enormous benefits of contrastive augmentation for document
representation learning with two additional insights: 1) including data
augmentation in a contrastive way can substantially improve the embedding
quality in unsupervised document representation learning, and 2) in general,
stochastic augmentations generated by simple word-level manipulation work much
better than sentence-level and document-level ones. We plug our method into a
classifier and compare it with a broad range of baseline methods on six
benchmark datasets. Our method can decrease the classification error rate by up
to 6.4% over the SOTA approaches on the document classification task, matching
or even surpassing fully-supervised methods.

negative:
Conformal prediction constructs a confidence set for an unobserved response
of a feature vector based on previous identically distributed and exchangeable
observations of responses and features. It has a coverage guarantee at any
nominal level without additional assumptions on their distribution. Its
computation deplorably requires a refitting procedure for all replacement
candidates of the target response. In regression settings, this corresponds to
an infinite number of model fits. Apart from relatively simple estimators that
can be written as pieces of linear function of the response, efficiently
computing such sets is difficult, and is still considered as an open problem.
We exploit the fact that, \emph{often}, conformal prediction sets are intervals
whose boundaries can be efficiently approximated by classical root-finding
algorithms. We investigate how this approach can overcome many limitations of
formerly used strategies; we discuss its complexity and drawbacks.

negative:
Conformal predictive systems are a recent modification of conformal
predictors that output, in regression problems, probability distributions for
labels of test observations rather than set predictions. The extra information
provided by conformal predictive systems may be useful, e.g., in decision
making problems. Conformal predictive systems inherit the relative
computational inefficiency of conformal predictors. In this paper we discuss
two computationally efficient versions of conformal predictive systems, which
we call split conformal predictive systems and cross-conformal predictive
systems. The main advantage of split conformal predictive systems is their
guaranteed validity, whereas for cross-conformal predictive systems validity
only holds empirically and in the absence of excessive randomization. The main
advantage of cross-conformal predictive systems is their greater predictive
efficiency.

negative:
We investigate the pertinence of methods from algebraic topology for text
data analysis. These methods enable the development of
mathematically-principled isometric-invariant mappings from a set of vectors to
a document embedding, which is stable with respect to the geometry of the
document in the selected metric space. In this work, we evaluate the utility of
these topology-based document representations in traditional NLP tasks,
specifically document clustering and sentiment classification. We find that the
embeddings do not benefit text analysis. In fact, performance is worse than
simple techniques like $\textit{tf-idf}$, indicating that the geometry of the
document does not provide enough variability for classification on the basis of
topic or sentiment in the chosen datasets.

negative:
This note introduces the method of cross-conformal prediction, which is a
hybrid of the methods of inductive conformal prediction and cross-validation,
and studies its validity and predictive efficiency empirically.

negative:
This paper proposes a new method of probabilistic prediction, which is based
on conformal prediction. The method is applied to the standard USPS data set
and gives encouraging results.

negative:
Text Mining is a field that aims at extracting information from textual data.
One of the challenges of such field of study comes from the pre-processing
stage in which a vector (and structured) representation should be extracted
from unstructured data. The common extraction creates large and sparse vectors
representing the importance of each term to a document. As such, this usually
leads to the curse-of-dimensionality that plagues most machine learning
algorithms. To cope with this issue, in this paper we propose a new supervised
feature extraction and reduction algorithm, named DCDistance, that creates
features based on the distance between a document to a representative of each
class label. As such, the proposed technique can reduce the features set in
more than 99% of the original set. Additionally, this algorithm was also
capable of improving the classification accuracy over a set of benchmark
datasets when compared to traditional and state-of-the-art features selection
algorithms.

negative:
Document AI, or Document Intelligence, is a relatively new research topic
that refers to the techniques for automatically reading, understanding, and
analyzing business documents. It is an important research direction for natural
language processing and computer vision. In recent years, the popularity of
deep learning technology has greatly advanced the development of Document AI,
such as document layout analysis, visual information extraction, document
visual question answering, document image classification, etc. This paper
briefly reviews some of the representative models, tasks, and benchmark
datasets. Furthermore, we also introduce early-stage heuristic rule-based
document analysis, statistical machine learning algorithms, and deep learning
approaches especially pre-training methods. Finally, we look into future
directions for Document AI research.

negative:
Recent studies emphasize the need of document context in human evaluation of
machine translations, but little research has been done on the impact of user
interfaces on annotator productivity and the reliability of assessments. In
this work, we compare human assessment data from the last two WMT evaluation
campaigns collected via two different methods for document-level evaluation.
Our analysis shows that a document-centric approach to evaluation where the
annotator is presented with the entire document context on a screen leads to
higher quality segment and document level assessments. It improves the
correlation between segment and document scores and increases inter-annotator
agreement for document scores but is considerably more time consuming for
annotators.

negative:
Neural networks with deep architectures have demonstrated significant
performance improvements in computer vision, speech recognition, and natural
language processing. The challenges in information retrieval (IR), however, are
different from these other application areas. A common form of IR involves
ranking of documents--or short passages--in response to keyword-based queries.
Effective IR systems must deal with query-document vocabulary mismatch problem,
by modeling relationships between different query and document terms and how
they indicate relevance. Models should also consider lexical matches when the
query contains rare terms--such as a person's name or a product model
number--not seen during training, and to avoid retrieving semantically related
but irrelevant results. In many real-life IR tasks, the retrieval involves
extremely large collections--such as the document index of a commercial Web
search engine--containing billions of documents. Efficient IR methods should
take advantage of specialized IR data structures, such as inverted index, to
efficiently retrieve from large collections. Given an information need, the IR
system also mediates how much exposure an information artifact receives by
deciding whether it should be displayed, and where it should be positioned,
among other results. Exposure-aware IR systems may optimize for additional
objectives, besides relevance, such as parity of exposure for retrieved items
and content publishers. In this thesis, we present novel neural architectures
and methods motivated by the specific needs and challenges of IR tasks.

negative:
This note discusses a simple modification of cross-conformal prediction
inspired by recent work on e-values. The precursor of conformal prediction
developed in the 1990s by Gammerman, Vapnik, and Vovk was also based on
e-values and is called conformal e-prediction in this note. Replacing e-values
by p-values led to conformal prediction, which has important advantages over
conformal e-prediction without obvious disadvantages. The situation with
cross-conformal prediction is, however, different: whereas for cross-conformal
prediction validity is only an empirical fact (and can be broken with excessive
randomization), this note draws the reader's attention to the obvious fact that
cross-conformal e-prediction enjoys a guaranteed property of validity.

negative:
We address the problem of solving mixed random linear equations. We have
unlabeled observations coming from multiple linear regressions, and each
observation corresponds to exactly one of the regression models. The goal is to
learn the linear regressors from the observations. Classically, Alternating
Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used
to solve this problem. AM iteratively alternates between the estimation of
labels and solving the regression problems with the estimated labels.
Empirically, it is observed that, for a large variety of non-convex problems
including mixed linear regression, AM converges at a much faster rate compared
to gradient based algorithms. However, the existing theory suggests similar
rate of convergence for AM and gradient based methods, failing to capture this
empirical behavior. In this paper, we close this gap between theory and
practice for the special case of a mixture of $2$ linear regressions. We show
that, provided initialized properly, AM enjoys a \emph{super-linear} rate of
convergence in certain parameter regimes. To the best of our knowledge, this is
the first work that theoretically establishes such rate for AM. Hence, if we
want to recover the unknown regressors upto an error (in $\ell_2$ norm) of
$\epsilon$, AM only takes $\mathcal{O}(\log \log (1/\epsilon))$ iterations.
Furthermore, we compare AM with a gradient based heuristic algorithm
empirically and show that AM dominates in iteration complexity as well as
wall-clock time.