query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I am currently developing a transparent and reliable AI chatbot. I am in search of AI models that can complete incomplete sentences or answer questions based on the next token prediction. For transparency, I need models that can explicitly compute the probability of the next token and update it in real-time based on user input. I also aim to provide a visualization that displays the perceptrons and weights utilized during each inference to the user. To ensure trustworthiness, I require techniques that can perturb the training set to increase the model's resilience against adversarial attacks. The model should also possess the ability to forget or replace outdated knowledge, as some data from the training set may become incorrect over time. Considering that a significant portion of potential training data from the internet includes sensitive personal information, I also require an efficient filter to automatically mask such information prior to its use in training. Ultimately, my objective is to develop an AI chatbot that is both transparent and trustworthy.

negative:
When recurrent neural network transducers (RNNTs) are trained using the
typical maximum likelihood criterion, the prediction network is trained only on
ground truth label sequences. This leads to a mismatch during inference, known
as exposure bias, when the model must deal with label sequences containing
errors. In this paper we investigate approaches to reducing exposure bias in
training to improve the generalization of RNNT models for automatic speech
recognition (ASR). A label-preserving input perturbation to the prediction
network is introduced. The input token sequences are perturbed using SwitchOut
and scheduled sampling based on an additional token language model. Experiments
conducted on the 300-hour Switchboard dataset demonstrate their effectiveness.
By reducing the exposure bias, we show that we can further improve the accuracy
of a high-performance RNNT ASR model and obtain state-of-the-art results on the
300-hour Switchboard dataset.

negative:
Recent neural models of dialogue generation offer great promise for
generating responses for conversational agents, but tend to be shortsighted,
predicting utterances one at a time while ignoring their influence on future
outcomes. Modeling the future direction of a dialogue is crucial to generating
coherent, interesting dialogues, a need which led traditional NLP models of
dialogue to draw on reinforcement learning. In this paper, we show how to
integrate these goals, applying deep reinforcement learning to model future
reward in chatbot dialogue. The model simulates dialogues between two virtual
agents, using policy gradient methods to reward sequences that display three
useful conversational properties: informativity (non-repetitive turns),
coherence, and ease of answering (related to forward-looking function). We
evaluate our model on diversity, length as well as with human judges, showing
that the proposed algorithm generates more interactive responses and manages to
foster a more sustained conversation in dialogue simulation. This work marks a
first step towards learning a neural conversational model based on the
long-term success of dialogues.

negative:
Recent developments have established the vulnerability of deep Reinforcement
Learning (RL) to policy manipulation attacks via adversarial perturbations. In
this paper, we investigate the robustness and resilience of deep RL to
training-time and test-time attacks. Through experimental results, we
demonstrate that under noncontiguous training-time attacks, Deep Q-Network
(DQN) agents can recover and adapt to the adversarial conditions by reactively
adjusting the policy. Our results also show that policies learned under
adversarial perturbations are more robust to test-time attacks. Furthermore, we
compare the performance of $\epsilon$-greedy and parameter-space noise
exploration methods in terms of robustness and resilience against adversarial
perturbations.

negative:
The Neural Machine Translation (NMT) model is essentially a joint language
model conditioned on both the source sentence and partial translation.
Therefore, the NMT model naturally involves the mechanism of the Language Model
(LM) that predicts the next token only based on partial translation. Despite
its success, NMT still suffers from the hallucination problem, generating
fluent but inadequate translations. The main reason is that NMT pays excessive
attention to the partial translation while neglecting the source sentence to
some extent, namely overconfidence of the LM. Accordingly, we define the Margin
between the NMT and the LM, calculated by subtracting the predicted probability
of the LM from that of the NMT model for each token. The Margin is negatively
correlated to the overconfidence degree of the LM. Based on the property, we
propose a Margin-based Token-level Objective (MTO) and a Margin-based
Sentencelevel Objective (MSO) to maximize the Margin for preventing the LM from
being overconfident. Experiments on WMT14 English-to-German, WMT19
Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate
the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improvements,
respectively, compared to the Transformer baseline. The human evaluation
further verifies that our approaches improve translation adequacy as well as
fluency.

negative:
Recent research in Visual Question Answering (VQA) has revealed
state-of-the-art models to be inconsistent in their understanding of the world
-- they answer seemingly difficult questions requiring reasoning correctly but
get simpler associated sub-questions wrong. These sub-questions pertain to
lower level visual concepts in the image that models ideally should understand
to be able to answer the higher level question correctly. To address this, we
first present a gradient-based interpretability approach to determine the
questions most strongly correlated with the reasoning question on an image, and
use this to evaluate VQA models on their ability to identify the relevant
sub-questions needed to answer a reasoning question. Next, we propose a
contrastive gradient learning based approach called Sub-question Oriented
Tuning (SOrT) which encourages models to rank relevant sub-questions higher
than irrelevant questions for an <image, reasoning-question> pair. We show that
SOrT improves model consistency by upto 6.5% points over existing baselines,
while also improving visual grounding.

negative:
Attention is sparse in vision transformers. We observe the final prediction
in vision transformers is only based on a subset of most informative tokens,
which is sufficient for accurate image recognition. Based on this observation,
we propose a dynamic token sparsification framework to prune redundant tokens
progressively and dynamically based on the input. Specifically, we devise a
lightweight prediction module to estimate the importance score of each token
given the current features. The module is added to different layers to prune
redundant tokens hierarchically. To optimize the prediction module in an
end-to-end manner, we propose an attention masking strategy to differentiably
prune a token by blocking its interactions with other tokens. Benefiting from
the nature of self-attention, the unstructured sparse tokens are still hardware
friendly, which makes our framework easy to achieve actual speed-up. By
hierarchically pruning 66% of the input tokens, our method greatly reduces
31%~37% FLOPs and improves the throughput by over 40% while the drop of
accuracy is within 0.5% for various vision transformers. Equipped with the
dynamic token sparsification framework, DynamicViT models can achieve very
competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs
and vision transformers on ImageNet. Code is available at
https://github.com/raoyongming/DynamicViT

negative:
To demystify the "black box" property of deep neural networks for natural
language processing (NLP), several methods have been proposed to interpret
their predictions by measuring the change in prediction probability after
erasing each token of an input. Since existing methods replace each token with
a predefined value (i.e., zero), the resulting sentence lies out of the
training data distribution, yielding misleading interpretations. In this study,
we raise the out-of-distribution problem induced by the existing interpretation
methods and present a remedy; we propose to marginalize each token out. We
interpret various NLP models trained for sentiment analysis and natural
language inference using the proposed method.

negative:
Recent deep learning models have shown remarkable performance in image
classification. While these deep learning systems are getting closer to
practical deployment, the common assumption made about data is that it does not
carry any sensitive information. This assumption may not hold for many
practical cases, especially in the domain where an individual's personal
information is involved, like healthcare and facial recognition systems. We
posit that selectively removing features in this latent space can protect the
sensitive information and provide a better privacy-utility trade-off.
Consequently, we propose DISCO which learns a dynamic and data driven pruning
filter to selectively obfuscate sensitive information in the feature space. We
propose diverse attack schemes for sensitive inputs \& attributes and
demonstrate the effectiveness of DISCO against state-of-the-art methods through
quantitative and qualitative evaluation. Finally, we also release an evaluation
benchmark dataset of 1 million sensitive representations to encourage rigorous
exploration of novel attack schemes.

negative:
We present two categories of model-agnostic adversarial strategies that
reveal the weaknesses of several generative, task-oriented dialogue models:
Should-Not-Change strategies that evaluate over-sensitivity to small and
semantics-preserving edits, as well as Should-Change strategies that test if a
model is over-stable against subtle yet semantics-changing modifications. We
next perform adversarial training with each strategy, employing a max-margin
approach for negative generative examples. This not only makes the target
dialogue model more robust to the adversarial inputs, but also helps it perform
significantly better on the original inputs. Moreover, training on all
strategies combined achieves further improvements, achieving a new
state-of-the-art performance on the original task (also verified via human
evaluation). In addition to adversarial training, we also address the
robustness task at the model-level, by feeding it subword units as both inputs
and outputs, and show that the resulting model is equally competitive, requires
only 1/4 of the original vocabulary size, and is robust to one of the
adversarial strategies (to which the original model is vulnerable) even without
adversarial training.

negative:
In complex processes, various events can happen in different sequences. The
prediction of the next event given an a-priori process state is of importance
in such processes. Recent methods have proposed deep learning techniques such
as recurrent neural networks, developed on raw event logs, to predict the next
event from a process state. However, such deep learning models by themselves
lack a clear representation of the process states. At the same time, recent
methods have neglected the time feature of event instances. In this paper, we
take advantage of Petri nets as a powerful tool in modeling complex process
behaviors considering time as an elemental variable. We propose an approach
which starts from a Petri net process model constructed by a process mining
algorithm. We enhance the Petri net model with time decay functions to create
continuous process state samples. Finally, we use these samples in combination
with discrete token movement counters and Petri net markings to train a deep
learning model that predicts the next event. We demonstrate significant
performance improvements and outperform the state-of-the-art methods on nine
real-world benchmark event logs.

negative:
Neural language models predict the next token using a latent representation
of the immediate token history. Recently, various methods for augmenting neural
language models with an attention mechanism over a differentiable memory have
been proposed. For predicting the next token, these models query information
from a memory of the recent history which can facilitate learning mid- and
long-range dependencies. However, conventional attention mechanisms used in
memory-augmented neural language models produce a single output vector per time
step. This vector is used both for predicting the next token as well as for the
key and value of a differentiable memory of a token history. In this paper, we
propose a neural language model with a key-value attention mechanism that
outputs separate representations for the key and value of a differentiable
memory, as well as for encoding the next-word distribution. This model
outperforms existing memory-augmented neural language models on two corpora.
Yet, we found that our method mainly utilizes a memory of the five most recent
output representations. This led to the unexpected main finding that a much
simpler model based only on the concatenation of recent output representations
from previous time steps is on par with more sophisticated memory-augmented
neural language models.

negative:
In hybrid human-AI systems, users need to decide whether or not to trust an
algorithmic prediction while the true error in the prediction is unknown. To
accommodate such settings, we introduce RETRO-VIZ, a method for (i) estimating
and (ii) explaining trustworthiness of regression predictions. It consists of
RETRO, a quantitative estimate of the trustworthiness of a prediction, and VIZ,
a visual explanation that helps users identify the reasons for the (lack of)
trustworthiness of a prediction. We find that RETRO-scores negatively correlate
with prediction error across 117 experimental settings, indicating that RETRO
provides a useful measure to distinguish trustworthy predictions from
untrustworthy ones. In a user study with 41 participants, we find that
VIZ-explanations help users identify whether a prediction is trustworthy or
not: on average, 95.1% of participants correctly select the more trustworthy
prediction, given a pair of predictions. In addition, an average of 75.6% of
participants can accurately describe why a prediction seems to be (not)
trustworthy. Finally, we find that the vast majority of users subjectively
experience RETRO-VIZ as a useful tool to assess the trustworthiness of
algorithmic predictions.

negative:
Despite widespread adoption, machine learning models remain mostly black
boxes. Understanding the reasons behind predictions is, however, quite
important in assessing trust, which is fundamental if one plans to take action
based on a prediction, or when choosing whether to deploy a new model. Such
understanding also provides insights into the model, which can be used to
transform an untrustworthy model or prediction into a trustworthy one. In this
work, we propose LIME, a novel explanation technique that explains the
predictions of any classifier in an interpretable and faithful manner, by
learning an interpretable model locally around the prediction. We also propose
a method to explain models by presenting representative individual predictions
and their explanations in a non-redundant way, framing the task as a submodular
optimization problem. We demonstrate the flexibility of these methods by
explaining different models for text (e.g. random forests) and image
classification (e.g. neural networks). We show the utility of explanations via
novel experiments, both simulated and with human subjects, on various scenarios
that require trust: deciding if one should trust a prediction, choosing between
models, improving an untrustworthy classifier, and identifying why a classifier
should not be trusted.

negative:
There have been many advances in the artificial intelligence field due to the
emergence of deep learning. In almost all sub-fields, artificial neural
networks have reached or exceeded human-level performance. However, most of the
models are not interpretable. As a result, it is hard to trust their decisions,
especially in life and death scenarios. In recent years, there has been a
movement toward creating explainable artificial intelligence, but most work to
date has concentrated on image processing models, as it is easier for humans to
perceive visual patterns. There has been little work in other fields like
natural language processing. In this paper, we train a convolutional model on
textual data and analyze the global logic of the model by studying its filter
values. In the end, we find the most important words in our corpus to our
models logic and remove the rest (95%). New models trained on just the 5% most
important words can achieve the same performance as the original model while
reducing training time by more than half. Approaches such as this will help us
to understand NLP models, explain their decisions according to their word
choices, and improve them by finding blind spots and biases.

negative:
Autonomous AI systems will be entering human society in the near future to
provide services and work alongside humans. For those systems to be accepted
and trusted, the users should be able to understand the reasoning process of
the system, i.e. the system should be transparent. System transparency enables
humans to form coherent explanations of the system's decisions and actions.
Transparency is important not only for user trust, but also for software
debugging and certification. In recent years, Deep Neural Networks have made
great advances in multiple application areas. However, deep neural networks are
opaque. In this paper, we report on work in transparency in Deep Reinforcement
Learning Networks (DRLN). Such networks have been extremely successful in
accurately learning action control in image input domains, such as Atari games.
In this paper, we propose a novel and general method that (a) incorporates
explicit object recognition processing into deep reinforcement learning models,
(b) forms the basis for the development of "object saliency maps", to provide
visualization of internal states of DRLNs, thus enabling the formation of
explanations and (c) can be incorporated in any existing deep reinforcement
learning framework. We present computational results and human experiments to
evaluate our approach.

negative:
Human mobility prediction is a core functionality in many location-based
services and applications. However, due to the sparsity of mobility data, it is
not an easy task to predict future POIs (place-of-interests) that are going to
be visited. In this paper, we propose MobTCast, a Transformer-based
context-aware network for mobility prediction. Specifically, we explore the
influence of four types of context in the mobility prediction: temporal,
semantic, social and geographical contexts. We first design a base mobility
feature extractor using the Transformer architecture, which takes both the
history POI sequence and the semantic information as input. It handles both the
temporal and semantic contexts. Based on the base extractor and the social
connections of a user, we employ a self-attention module to model the influence
of the social context. Furthermore, unlike existing methods, we introduce a
location prediction branch in MobTCast as an auxiliary task to model the
geographical context and predict the next location. Intuitively, the
geographical distance between the location of the predicted POI and the
predicted location from the auxiliary branch should be as close as possible. To
reflect this relation, we design a consistency loss to further improve the POI
prediction performance. In our experimental results, MobTCast outperforms other
state-of-the-art next POI prediction methods. Our approach illustrates the
value of including different types of context in next POI prediction.

negative:
Attention-based encoder-decoder (AED) models have achieved promising
performance in speech recognition. However, because the decoder predicts text
tokens (such as characters or words) in an autoregressive manner, it is
difficult for an AED model to predict all tokens in parallel. This makes the
inference speed relatively slow. We believe that because the encoder already
captures the whole speech utterance, which has the token-level relationship
implicitly, we can predict a token without explicitly autoregressive language
modeling. When the prediction of a token does not rely on other tokens, the
parallel prediction of all tokens in the sequence is realizable. Based on this
idea, we propose a non-autoregressive speech recognition model called LASO
(Listen Attentively, and Spell Once). The model consists of an encoder, a
decoder, and a position dependent summarizer (PDS). The three modules are based
on basic attention blocks. The encoder extracts high-level representations from
the speech. The PDS uses positional encodings corresponding to tokens to
convert the acoustic representations into token-level representations. The
decoder further captures token-level relationships with the self-attention
mechanism. At last, the probability distribution on the vocabulary is computed
for each token position. Therefore, speech recognition is re-formulated as a
position-wise classification problem. Further, we propose a cross-modal
transfer learning method to refine semantics from a large-scale pre-trained
language model BERT for improving the performance.

negative:
Personal assistant AI systems such as Siri, Cortana, and Alexa have become
widely used as a means to accomplish tasks through natural language commands.
However, components in these systems generally rely on supervised machine
learning algorithms that require large amounts of hand-annotated training data,
which is expensive and time consuming to collect. The ability to incorporate
unsupervised, weakly supervised, or distantly supervised data holds significant
promise in overcoming this bottleneck. In this paper, we describe a framework
that leverages user engagement signals (user behaviors that demonstrate a
positive or negative response to content) to automatically create granular
entity labels for training data augmentation. Strategies such as multi-task
learning and validation using an external knowledge base are employed to
incorporate the engagement annotated data and to boost the model's accuracy on
a sequence labeling task. Our results show that learning from data
automatically labeled by user engagement signals achieves significant accuracy
gains in a production deep learning system, when measured on both the sequence
labeling task as well as on user facing results produced by the system
end-to-end. We believe this is the first use of user engagement signals to help
generate training data for a sequence labeling task on a large scale, and can
be applied in practical settings to speed up new feature deployment when little
human annotated data is available.

negative:
Recently, large pre-trained neural language models have attained remarkable
performance on many downstream natural language processing (NLP) applications
via fine-tuning. In this paper, we target at how to further improve the token
representations on the language models. We, therefore, propose a simple yet
effective plug-and-play module, Sequential Attention Module (SAM), on the token
embeddings learned from a pre-trained language model. Our proposed SAM consists
of two main attention modules deployed sequentially: Feature-wise Attention
Module (FAM) and Token-wise Attention Module (TAM). More specifically, FAM can
effectively identify the importance of features at each dimension and promote
the effect via dot-product on the original token embeddings for downstream NLP
applications. Meanwhile, TAM can further re-weight the features at the
token-wise level. Moreover, we propose an adaptive filter on FAM to prevent
noise impact and increase information absorption. Finally, we conduct extensive
experiments to demonstrate the advantages and properties of our proposed SAM.
We first show how SAM plays a primary role in the champion solution of two
subtasks of SemEval'21 Task 7. After that, we apply SAM on sentiment analysis
and three popular NLP tasks and demonstrate that SAM consistently outperforms
the state-of-the-art baselines.

negative:
Masked language modeling (MLM), a self-supervised pretraining objective, is
widely used in natural language processing for learning text representations.
MLM trains a model to predict a random sample of input tokens that have been
replaced by a [MASK] placeholder in a multi-class setting over the entire
vocabulary. When pretraining, it is common to use alongside MLM other auxiliary
objectives on the token or sequence level to improve downstream performance
(e.g. next sentence prediction). However, no previous work so far has attempted
in examining whether other simpler linguistically intuitive or not objectives
can be used standalone as main pretraining objectives. In this paper, we
explore five simple pretraining objectives based on token-level classification
tasks as replacements of MLM. Empirical results on GLUE and SQuAD show that our
proposed methods achieve comparable or better performance to MLM using a
BERT-BASE architecture. We further validate our methods using smaller models,
showing that pretraining a model with 41% of the BERT-BASE's parameters,
BERT-MEDIUM results in only a 1% drop in GLUE scores with our best objective.

negative:
We introduce a novel approach to transformers that learns hierarchical
representations in multiparty dialogue. First, three language modeling tasks
are used to pre-train the transformers, token- and utterance-level language
modeling and utterance order prediction, that learn both token and utterance
embeddings for better understanding in dialogue contexts. Then, multi-task
learning between the utterance prediction and the token span prediction is
applied to fine-tune for span-based question answering (QA). Our approach is
evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over
the two state-of-the-art transformer models, BERT and RoBERTa, respectively.

negative:
This paper describes problems in AI research and how the SP System (described
in an appendix) may help to solve them. Most of the problems are described by
leading researchers in AI in interviews with science writer Martin Ford, and
reported by him in his book {\em Architects of Intelligence}. These problems
are: the need to bridge the divide between symbolic and non-symbolic kinds of
knowledge and processing; the tendency of deep neural networks (DNNs) to make
large and unexpected errors in recognition; the need to strengthen the
representation and processing of natural languages; the challenges of
unsupervised learning; the need for a coherent account of generalisation; how
to learn usable knowledge from a single exposure; how to achieve transfer
learning; how to increase the efficiency of AI processing; the need for
transparency in AI structures and processes; how to achieve varieties of
probabilistic reasoning; the need for more emphasis on top-down strategies; how
to minimise the risk of accidents with self-driving vehicles; the need for
strong compositionality in AI knowledge; the challenges of commonsense
reasoning and commonsense knowledge; establishing the importance of information
compression in AI research; establishing the importance of a biological
perspective in AI research; establishing whether knowledge in the brain is
represented in `distributed' or `localist' form; how to bypassing the limited
scope for adaptation in deep neural networks; the need to develop `broad AI';
and how to eliminate the problem of catastrophic forgetting.

negative:
Traffic state prediction is necessary for many Intelligent Transportation
Systems applications. Recent developments of the topic have focused on
network-wide, multi-step prediction, where state of the art performance is
achieved via deep learning models, in particular, graph neural network-based
models. While the prediction accuracy of deep learning models is high, these
models' robustness has raised many safety concerns, given that imperceptible
perturbations added to input can substantially degrade the model performance.
In this work, we propose an adversarial attack framework by treating the
prediction model as a black-box, i.e., assuming no knowledge of the model
architecture, training data, and (hyper)parameters. However, we assume that the
adversary can oracle the prediction model with any input and obtain
corresponding output. Next, the adversary can train a substitute model using
input-output pairs and generate adversarial signals based on the substitute
model. To test the attack effectiveness, two state of the art, graph neural
network-based models (GCGRNN and DCRNN) are examined. As a result, the
adversary can degrade the target model's prediction accuracy up to $54\%$. In
comparison, two conventional statistical models (linear regression and
historical average) are also examined. While these two models do not produce
high prediction accuracy, they are either influenced negligibly (less than
$3\%$) or are immune to the adversary's attack.

negative:
The increased adoption of Artificial Intelligence (AI) presents an
opportunity to solve many socio-economic and environmental challenges; however,
this cannot happen without securing AI-enabled technologies. In recent years,
most AI models are vulnerable to advanced and sophisticated hacking techniques.
This challenge has motivated concerted research efforts into adversarial AI,
with the aim of developing robust machine and deep learning models that are
resilient to different types of adversarial scenarios. In this paper, we
present a holistic cyber security review that demonstrates adversarial attacks
against AI applications, including aspects such as adversarial knowledge and
capabilities, as well as existing methods for generating adversarial examples
and existing cyber defence models. We explain mathematical AI models,
especially new variants of reinforcement and federated learning, to demonstrate
how attack vectors would exploit vulnerabilities of AI models. We also propose
a systematic framework for demonstrating attack techniques against AI
applications and reviewed several cyber defences that would protect AI
applications against those attacks. We also highlight the importance of
understanding the adversarial goals and their capabilities, especially the
recent attacks against industry applications, to develop adaptive defences that
assess to secure AI applications. Finally, we describe the main challenges and
future research directions in the domain of security and privacy of AI
technologies.

negative:
Video captioning is a challenging task that captures different visual parts
and describes them in sentences, for it requires visual and linguistic
coherence. The attention mechanism in the current video captioning method
learns to assign weight to each frame, promoting the decoder dynamically. This
may not explicitly model the correlation and the temporal coherence of the
visual features extracted in the sequence frames.To generate semantically
coherent sentences, we propose a new Visual-aware Attention (VA) model, which
concatenates dynamic changes of temporal sequence frames with the words at the
previous moment, as the input of attention mechanism to extract sequence
features.In addition, the prevalent approaches widely use the teacher-forcing
(TF) learning during training, where the next token is generated conditioned on
the previous ground-truth tokens. The semantic information in the previously
generated tokens is lost. Therefore, we design a self-forcing (SF) stream that
takes the semantic information in the probability distribution of the previous
token as input to enhance the current token.The Dual-stream Decoder (DD)
architecture unifies the TF and SF streams, generating sentences to promote the
annotated captioning for both streams.Meanwhile, with the Dual-stream Decoder
utilized, the exposure bias problem is alleviated, caused by the discrepancy
between the training and testing in the TF learning.The effectiveness of the
proposed Visual-aware Attention Dual-stream Decoder (VADD) is demonstrated
through the result of experimental studies on Microsoft video description
(MSVD) corpus and MSR-Video to text (MSR-VTT) datasets.

negative:
We develop a chatbot using Deep Bidirectional Transformer models (BERT) to
handle client questions in financial investment customer service. The bot can
recognize 381 intents, and decides when to say "I don't know" and escalates
irrelevant/uncertain questions to human operators. Our main novel contribution
is the discussion about uncertainty measure for BERT, where three different
approaches are systematically compared on real problems. We investigated two
uncertainty metrics, information entropy and variance of dropout sampling in
BERT, followed by mixed-integer programming to optimize decision thresholds.
Another novel contribution is the usage of BERT as a language model in
automatic spelling correction. Inputs with accidental spelling errors can
significantly decrease intent classification performance. The proposed approach
combines probabilities from masked language model and word edit distances to
find the best corrections for misspelled words. The chatbot and the entire
conversational AI system are developed using open-source tools, and deployed
within our company's intranet. The proposed approach can be useful for
industries seeking similar in-house solutions in their specific business
domains. We share all our code and a sample chatbot built on a public dataset
on Github.

negative:
While improving neural dialogue agents' factual accuracy is the object of
much research, another important aspect of communication, less studied in the
setting of neural dialogue, is transparency about ignorance. In this work, we
analyze to what extent state-of-the-art chit-chat models are linguistically
calibrated in the sense that their verbalized expression of doubt (or
confidence) matches the likelihood that the model's responses are factually
incorrect (or correct). We find that these models are poorly calibrated, yet we
show that likelihood of correctness can accurately be predicted. By
incorporating such metacognitive features into the training of a controllable
generation model, we obtain a dialogue agent with greatly improved linguistic
calibration. While improving neural dialogue agents' factual accuracy is the
object of much research, another important aspect of communication, less
studied in the setting of neural dialogue, is transparency about ignorance. In
this work, we analyze to what extent state-of-the-art chit-chat models are
linguistically calibrated in the sense that their verbalized expression of
doubt (or confidence) matches the likelihood that the model's responses are
factually incorrect (or correct). We find that these models are poorly
calibrated, yet we show that likelihood of correctness can accurately be
predicted. By incorporating such metacognitive features into the training of a
controllable generation model, we obtain a dialogue agent with greatly improved
linguistic calibration.

negative:
Today, most large-scale conversational AI agents (e.g. Alexa, Siri, or Google
Assistant) are built using manually annotated data to train the different
components of the system. Typically, the accuracy of the ML models in these
components are improved by manually transcribing and annotating data. As the
scope of these systems increase to cover more scenarios and domains, manual
annotation to improve the accuracy of these components becomes prohibitively
costly and time consuming. In this paper, we propose a system that leverages
user-system interaction feedback signals to automate learning without any
manual annotation. Users here tend to modify a previous query in hopes of
fixing an error in the previous turn to get the right results. These
reformulations, which are often preceded by defective experiences caused by
errors in ASR, NLU, ER or the application. In some cases, users may not
properly formulate their requests (e.g. providing partial title of a song), but
gleaning across a wider pool of users and sessions reveals the underlying
recurrent patterns. Our proposed self-learning system automatically detects the
errors, generate reformulations and deploys fixes to the runtime system to
correct different types of errors occurring in different components of the
system. In particular, we propose leveraging an absorbing Markov Chain model as
a collaborative filtering mechanism in a novel attempt to mine these patterns.
We show that our approach is highly scalable, and able to learn reformulations
that reduce Alexa-user errors by pooling anonymized data across millions of
customers. The proposed self-learning system achieves a win/loss ratio of 11.8
and effectively reduces the defect rate by more than 30% on utterance level
reformulations in our production A/B tests. To the best of our knowledge, this
is the first self-learning large-scale conversational AI system in production.

negative:
Our project task was to create a model that, given a speaker ID, chat
history, and an utterance query, can predict the response utterance in a
conversation. The model is personalized for each speaker. This task can be a
useful tool for building speech bots that talk in a human-like manner in a live
conversation. Further, we succeeded at using dense-vector encoding clustering
to be able to retrieve relevant historical dialogue context, a useful strategy
for overcoming the input limitations of neural-based models when predictions
require longer-term references from the dialogue history. In this paper, we
have implemented a state-of-the-art model using pre-training and fine-tuning
techniques built on transformer architecture and multi-headed attention blocks
for the Switchboard corpus. We also show how efficient vector clustering
algorithms can be used for real-time utterance predictions that require no
training and therefore work on offline and encrypted message histories.

negative:
Current conversational systems can follow simple commands and answer basic
questions, but they have difficulty maintaining coherent and open-ended
conversations about specific topics. Competitions like the Conversational
Intelligence (ConvAI) challenge are being organized to push the research
development towards that goal. This article presents in detail the RLLChatbot
that participated in the 2017 ConvAI challenge. The goal of this research is to
better understand how current deep learning and reinforcement learning tools
can be used to build a robust yet flexible open domain conversational agent. We
provide a thorough description of how a dialog system can be built and trained
from mostly public-domain datasets using an ensemble model. The first
contribution of this work is a detailed description and analysis of different
text generation models in addition to novel message ranking and selection
methods. Moreover, a new open-source conversational dataset is presented.
Training on this data significantly improves the Recall@k score of the ranking
and selection mechanisms compared to our baseline model responsible for
selecting the message returned at each interaction.

negative:
Sentence order prediction is the task of finding the correct order of
sentences in a randomly ordered document. Correctly ordering the sentences
requires an understanding of coherence with respect to the chronological
sequence of events described in the text. Document-level contextual
understanding and commonsense knowledge centered around these events are often
essential in uncovering this coherence and predicting the exact chronological
order. In this paper, we introduce STaCK -- a framework based on graph neural
networks and temporal commonsense knowledge to model global information and
predict the relative order of sentences. Our graph network accumulates temporal
evidence using knowledge of `past' and `future' and formulates sentence
ordering as a constrained edge classification problem. We report results on
five different datasets, and empirically show that the proposed method is
naturally suitable for order prediction. The implementation of this work is
publicly available at: https://github.com/declare-lab/sentence-ordering.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
Highly regularized LSTMs achieve impressive results on several benchmark
datasets in language modeling. We propose a new regularization method based on
decoding the last token in the context using the predicted distribution of the
next token. This biases the model towards retaining more contextual
information, in turn improving its ability to predict the next token. With
negligible overhead in the number of parameters and training time, our Past
Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on
the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax.
We also show gains by using PDR in combination with a mixture-of-softmaxes,
achieving a word level perplexity of 53.8 and 60.5 on these datasets. In
addition, our method achieves 1.169 bits-per-character on the Penn Treebank
Character dataset for character level language modeling. These results
constitute a new state-of-the-art in their respective settings.

negative:
Token embeddings in multilingual BERT (m-BERT) contain both language and
semantic information. We find that the representation of a language can be
obtained by simply averaging the embeddings of the tokens of the language.
Given this language representation, we control the output languages of
multilingual BERT by manipulating the token embeddings, thus achieving
unsupervised token translation. We further propose a computationally cheap but
effective approach to improve the cross-lingual ability of m-BERT based on this
observation.

negative:
We propose a new model, DocHopper, that iteratively attends to different
parts of long, hierarchically structured documents to answer complex questions.
Similar to multi-hop question-answering (QA) systems, at each step, DocHopper
uses a query $q$ to attend to information from a document, combines this
``retrieved'' information with $q$ to produce the next query. However, in
contrast to most previous multi-hop QA systems, DocHopper is able to
``retrieve'' either short passages or long sections of the document, thus
emulating a multi-step process of ``navigating'' through a long document to
answer a question. To enable this novel behavior, DocHopper does not combine
document information with $q$ by concatenating text to the text of $q$, but by
combining a compact neural representation of $q$ with a compact neural
representation of a hierarchical part of the document, which can potentially be
quite large. We experiment with DocHopper on four different QA tasks that
require reading long and complex documents to answer multi-hop questions, and
show that DocHopper achieves state-of-the-art results on three of the datasets.
Additionally, DocHopper is efficient at inference time, being 3--10 times
faster than the baselines.

negative:
The rapid development of Artificial Intelligence (AI) technology has enabled
the deployment of various systems based on it. However, many current AI systems
are found vulnerable to imperceptible attacks, biased against underrepresented
groups, lacking in user privacy protection. These shortcomings degrade user
experience and erode people's trust in all AI systems. In this review, we
provide AI practitioners with a comprehensive guide for building trustworthy AI
systems. We first introduce the theoretical framework of important aspects of
AI trustworthiness, including robustness, generalization, explainability,
transparency, reproducibility, fairness, privacy preservation, and
accountability. To unify currently available but fragmented approaches toward
trustworthy AI, we organize them in a systematic approach that considers the
entire lifecycle of AI systems, ranging from data acquisition to model
development, to system development and deployment, finally to continuous
monitoring and governance. In this framework, we offer concrete action items
for practitioners and societal stakeholders (e.g., researchers, engineers, and
regulators) to improve AI trustworthiness. Finally, we identify key
opportunities and challenges for the future development of trustworthy AI
systems, where we identify the need for a paradigm shift toward comprehensively
trustworthy AI systems.

negative:
Recent advances in deep neural networks, language modeling and language
generation have introduced new ideas to the field of conversational agents. As
a result, deep neural models such as sequence-to-sequence, Memory Networks, and
the Transformer have become key ingredients of state-of-the-art dialog systems.
While those models are able to generate meaningful responses even in unseen
situation, they need a lot of training data to build a reliable model. Thus,
most real-world systems stuck to traditional approaches based on information
retrieval and even hand-crafted rules, due to their robustness and
effectiveness, especially for narrow-focused conversations. Here, we present a
method that adapts a deep neural architecture from the domain of machine
reading comprehension to re-rank the suggested answers from different models
using the question as context. We train our model using negative sampling based
on question-answer pairs from the Twitter Customer Support Dataset.The
experimental results show that our re-ranking framework can improve the
performance in terms of word overlap and semantics both for individual models
as well as for model combinations.

negative:
The ability of a machine to communicate with humans has long been associated
with the general success of AI. This dates back to Alan Turing's epoch-making
work in the early 1950s, which proposes that a machine's intelligence can be
tested by how well it, the machine, can fool a human into believing that the
machine is a human through dialogue conversations. Many systems learn
generation rules from a minimal set of authored rules or labels on top of
hand-coded rules or templates, and thus are both expensive and difficult to
extend to open-domain scenarios. Recently, the emergence of neural network
models the potential to solve many of the problems in dialogue learning that
earlier systems cannot tackle: the end-to-end neural frameworks offer the
promise of scalability and language-independence, together with the ability to
track the dialogue state and then mapping between states and dialogue actions
in a way not possible with conventional systems. On the other hand, neural
systems bring about new challenges: they tend to output dull and generic
responses; they lack a consistent or a coherent persona; they are usually
optimized through single-turn conversations and are incapable of handling the
long-term success of a conversation; and they are not able to take the
advantage of the interactions with humans. This dissertation attempts to tackle
these challenges: Contributions are two-fold: (1) we address new challenges
presented by neural network models in open-domain dialogue generation systems;
(2) we develop interactive question-answering dialogue systems by (a) giving
the agent the ability to ask questions and (b) training a conversation agent
through interactions with humans in an online fashion, where a bot improves
through communicating with humans and learning from the mistakes that it makes.

negative:
The encoder-decoder framework achieves state-of-the-art results in keyphrase
generation (KG) tasks by predicting both present keyphrases that appear in the
source document and absent keyphrases that do not. However, relying solely on
the source document can result in generating uncontrollable and inaccurate
absent keyphrases. To address these problems, we propose a novel graph-based
method that can capture explicit knowledge from related references. Our model
first retrieves some document-keyphrases pairs similar to the source document
from a pre-defined index as references. Then a heterogeneous graph is
constructed to capture relationships of different granularities between the
source document and its references. To guide the decoding process, a
hierarchical attention and copy mechanism is introduced, which directly copies
appropriate words from both the source document and its references based on
their relevance and significance. The experimental results on multiple KG
benchmarks show that the proposed model achieves significant improvements
against other baseline models, especially with regard to the absent keyphrase
prediction.

negative:
In this work, we present the Chatbot Interaction with Artificial Intelligence
(CI-AI) framework as an approach to the training of deep learning chatbots for
task classification. The intelligent system augments human-sourced data via
artificial paraphrasing in order to generate a large set of training data for
further classical, attention, and language transformation-based learning
approaches for Natural Language Processing. Human beings are asked to
paraphrase commands and questions for task identification for further execution
of a machine. The commands and questions are split into training and validation
sets. A total of 483 responses were recorded. Secondly, the training set is
paraphrased by the T5 model in order to augment it with further data. Seven
state-of-the-art transformer-based text classification algorithms (BERT,
DistilBERT, RoBERTa, DistilRoBERTa, XLM, XLM-RoBERTa, and XLNet) are
benchmarked for both sets after fine-tuning on the training data for two
epochs. We find that all models are improved when training data is augmented by
the T5 model, with an average increase of classification accuracy by 4.01%. The
best result was the RoBERTa model trained on T5 augmented data which achieved
98.96% classification accuracy. Finally, we found that an ensemble of the five
best-performing transformer models via Logistic Regression of output label
predictions led to an accuracy of 99.59% on the dataset of human responses. A
highly-performing model allows the intelligent system to interpret human
commands at the social-interaction level through a chatbot-like interface (e.g.
"Robot, can we have a conversation?") and allows for better accessibility to AI
by non-technical users.

negative:
Artificial Intelligence (AI) is increasingly used in critical applications.
Thus, the need for dependable AI systems is rapidly growing. In 2018, the
European Commission appointed experts to a High-Level Expert Group on AI
(AI-HLEG). AI-HLEG defined Trustworthy AI as 1) lawful, 2) ethical, and 3)
robust and specified seven corresponding key requirements. To help development
organizations, AI-HLEG recently published the Assessment List for Trustworthy
AI (ALTAI). We present an illustrative case study from applying ALTAI to an
ongoing development project of an Advanced Driver-Assistance System (ADAS) that
relies on Machine Learning (ML). Our experience shows that ALTAI is largely
applicable to ADAS development, but specific parts related to human agency and
transparency can be disregarded. Moreover, bigger questions related to societal
and environmental impact cannot be tackled by an ADAS supplier in isolation. We
present how we plan to develop the ADAS to ensure ALTAI-compliance. Finally, we
provide three recommendations for the next revision of ALTAI, i.e., life-cycle
variants, domain-specific adaptations, and removed redundancy.

negative:
Despite the widespread use of Knowledge Graph Embeddings (KGE), little is
known about the security vulnerabilities that might disrupt their intended
behaviour. We study data poisoning attacks against KGE models for link
prediction. These attacks craft adversarial additions or deletions at training
time to cause model failure at test time. To select adversarial deletions, we
propose to use the model-agnostic instance attribution methods from
Interpretable Machine Learning, which identify the training instances that are
most influential to a neural model's predictions on test instances. We use
these influential triples as adversarial deletions. We further propose a
heuristic method to replace one of the two entities in each influential triple
to generate adversarial additions. Our experiments show that the proposed
strategies outperform the state-of-art data poisoning attacks on KGE models and
improve the MRR degradation due to the attacks by up to 62% over the baselines.

negative:
Cross-lingual document representations enable language understanding in
multilingual contexts and allow transfer learning from high-resource to
low-resource languages at the document level. Recently large pre-trained
language models such as BERT, XLM and XLM-RoBERTa have achieved great success
when fine-tuned on sentence-level downstream tasks. It is tempting to apply
these cross-lingual models to document representation learning. However, there
are two challenges: (1) these models impose high costs on long document
processing and thus many of them have strict length limit; (2) model
fine-tuning requires extra data and computational resources, which is not
practical in resource-limited settings. In this work, we address these
challenges by proposing unsupervised Language-Agnostic Weighted Document
Representations (LAWDR). We study the geometry of pre-trained sentence
embeddings and leverage it to derive document representations without
fine-tuning. Evaluated on cross-lingual document alignment, LAWDR demonstrates
comparable performance to state-of-the-art models on benchmark datasets.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
Explaining the predictions of AI models is paramount in safety-critical
applications, such as in legal or medical domains. One form of explanation for
a prediction is an extractive rationale, i.e., a subset of features of an
instance that lead the model to give its prediction on the instance. Previous
works on generating extractive rationales usually employ a two-phase model: a
selector that selects the most important features (i.e., the rationale)
followed by a predictor that makes the prediction based exclusively on the
selected features. One disadvantage of these works is that the main signal for
learning to select features comes from the comparison of the answers given by
the predictor and the ground-truth answers. In this work, we propose to squeeze
more information from the predictor via an information calibration method. More
precisely, we train two models jointly: one is a typical neural model that
solves the task at hand in an accurate but black-box manner, and the other is a
selector-predictor model that additionally produces a rationale for its
prediction. The first model is used as a guide to the second model. We use an
adversarial-based technique to calibrate the information extracted by the two
models such that the difference between them is an indicator of the missed or
over-selected features. In addition, for natural language tasks, we propose to
use a language-model-based regularizer to encourage the extraction of fluent
rationales. Experimental results on a sentiment analysis task as well as on
three tasks from the legal domain show the effectiveness of our approach to
rationale extraction.

negative:
Models are used in both Software Engineering (SE) and Artificial Intelligence
(AI). SE models may specify the architecture at different levels of abstraction
and for addressing different concerns at various stages of the software
development life-cycle, from early conceptualization and design, to
verification, implementation, testing and evolution. However, AI models may
provide smart capabilities, such as prediction and decision-making support. For
instance, in Machine Learning (ML), which is currently the most popular
sub-discipline of AI, mathematical models may learn useful patterns in the
observed data and can become capable of making predictions. The goal of this
work is to create synergy by bringing models in the said communities together
and proposing a holistic approach to model-driven software development for
intelligent systems that require ML. We illustrate how software models can
become capable of creating and dealing with ML models in a seamless manner. The
main focus is on the domain of the Internet of Things (IoT), where both ML and
model-driven SE play a key role. In the context of the need to take a
Cyber-Physical System-of-Systems perspective of the targeted architecture, an
integrated design environment for both SE and ML sub-systems would best support
the optimization and overall efficiency of the implementation of the resulting
system. In particular, we implement the proposed approach, called ML-Quadrat,
based on ThingML, and validate it using a case study from the IoT domain, as
well as through an empirical user evaluation. It transpires that the proposed
approach is not only feasible, but may also contribute to the performance leap
of software development for smart Cyber-Physical Systems (CPS) which are
connected to the IoT, as well as an enhanced user experience of the
practitioners who use the proposed modeling solution.

negative:
This paper summarizes our entries to both subtasks of the first DialDoc
shared task which focuses on the agent response prediction task in
goal-oriented document-grounded dialogs. The task is split into two subtasks:
predicting a span in a document that grounds an agent turn and generating an
agent response based on a dialog and grounding document. In the first subtask,
we restrict the set of valid spans to the ones defined in the dataset, use a
biaffine classifier to model spans, and finally use an ensemble of different
models. For the second subtask, we use a cascaded model which grounds the
response prediction on the predicted span instead of the full document. With
these approaches, we obtain significant improvements in both subtasks compared
to the baseline.

negative:
The history of AI has included several "waves" of ideas. The first wave, from
the mid-1950s to the 1980s, focused on logic and symbolic hand-encoded
representations of knowledge, the foundations of so-called "expert systems".
The second wave, starting in the 1990s, focused on statistics and machine
learning, in which, instead of hand-programming rules for behavior, programmers
constructed "statistical learning algorithms" that could be trained on large
datasets. In the most recent wave research in AI has largely focused on deep
(i.e., many-layered) neural networks, which are loosely inspired by the brain
and trained by "deep learning" methods. However, while deep neural networks
have led to many successes and new capabilities in computer vision, speech
recognition, language processing, game-playing, and robotics, their potential
for broad application remains limited by several factors.
  A concerning limitation is that even the most successful of today's AI
systems suffer from brittleness-they can fail in unexpected ways when faced
with situations that differ sufficiently from ones they have been trained on.
This lack of robustness also appears in the vulnerability of AI systems to
adversarial attacks, in which an adversary can subtly manipulate data in a way
to guarantee a specific wrong answer or action from an AI system. AI systems
also can absorb biases-based on gender, race, or other factors-from their
training data and further magnify these biases in their subsequent
decision-making. Taken together, these various limitations have prevented AI
systems such as automatic medical diagnosis or autonomous vehicles from being
sufficiently trustworthy for wide deployment. The massive proliferation of AI
across society will require radically new ideas to yield technology that will
not sacrifice our productivity, our quality of life, or our values.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
Next location prediction is of great importance for many location-based
applications and provides essential intelligence to business and governments.
In existing studies, a common approach to next location prediction is to learn
the sequential transitions with massive historical trajectories based on
conditional probability. Unfortunately, due to the time and space complexity,
these methods (e.g., Markov models) only use the just passed locations to
predict next locations, without considering all the passed locations in the
trajectory. In this paper, we seek to enhance the prediction performance by
considering the travel time from all the passed locations in the query
trajectory to a candidate next location. In particular, we propose a novel
method, called Travel Time Difference Model (TTDM), which exploits the
difference between the shortest travel time and the actual travel time to
predict next locations. Further, we integrate the TTDM with a Markov model via
a linear interpolation to yield a joint model, which computes the probability
of reaching each possible next location and returns the top-rankings as
results. We have conducted extensive experiments on two real datasets: the
vehicle passage record (VPR) data and the taxi trajectory data. The
experimental results demonstrate significant improvements in prediction
accuracy over existing solutions. For example, compared with the Markov model,
the top-1 accuracy improves by 40% on the VPR data and by 15.6% on the Taxi
data.

negative:
As a new type of e-commerce platform developed in recent years, local
consumer service platform provides users with software to consume service to
the nearby store or to the home, such as Groupon and Koubei. Different from
other common e-commerce platforms, the behavior of users on the local consumer
service platform is closely related to their real-time local context
information. Therefore, building a context-aware user behavior prediction
system is able to provide both merchants and users better service in local
consumer service platforms. However, most of the previous work just treats the
contextual information as an ordinary feature into the prediction model to
obtain the prediction list under a specific context, which ignores the fact
that the interest of a user in different contexts is often significantly
different. Hence, in this paper, we propose a context-aware heterogeneous graph
attention network (CHGAT) to dynamically generate the representation of the
user and to estimate the probability for future behavior. Specifically, we
first construct the meta-path based heterogeneous graphs with the historical
behaviors from multiple sources and comprehend heterogeneous vertices in the
graph with a novel unified knowledge representing approach. Next, a multi-level
attention mechanism is introduced for context-aware aggregation with graph
vertices, which contains the vertex-level attention network and the path-level
attention network. Both of them aim to capture the semantic correlation between
information contained in the graph and the outside real-time contextual
information in the search system. Then the model proposed in this paper
aggregates specific graphs with their corresponding context features and
obtains the representation of user interest under a specific context and input
it into the prediction network to finally obtain the predicted probability of
user behavior.

negative:
Providing accurate predictions is challenging for machine learning algorithms
when the number of features is larger than the number of samples in the data.
Prior knowledge can improve machine learning models by indicating relevant
variables and parameter values. Yet, this prior knowledge is often tacit and
only available from domain experts. We present a novel approach that uses
interactive visualization to elicit the tacit prior knowledge and uses it to
improve the accuracy of prediction models. The main component of our approach
is a user model that models the domain expert's knowledge of the relevance of
different features for a prediction task. In particular, based on the expert's
earlier input, the user model guides the selection of the features on which to
elicit user's knowledge next. The results of a controlled user study show that
the user model significantly improves prior knowledge elicitation and
prediction accuracy, when predicting the relative citation counts of scientific
documents in a specific domain.

negative:
In this work we present a formal theoretical framework for assessing and
analyzing two classes of malevolent action towards generic Artificial
Intelligence (AI) systems. Our results apply to general multi-class classifiers
that map from an input space into a decision space, including artificial neural
networks used in deep learning applications. Two classes of attacks are
considered. The first class involves adversarial examples and concerns the
introduction of small perturbations of the input data that cause
misclassification. The second class, introduced here for the first time and
named stealth attacks, involves small perturbations to the AI system itself.
Here the perturbed system produces whatever output is desired by the attacker
on a specific small data set, perhaps even a single input, but performs as
normal on a validation set (which is unknown to the attacker). We show that in
both cases, i.e., in the case of an attack based on adversarial examples and in
the case of a stealth attack, the dimensionality of the AI's decision-making
space is a major contributor to the AI's susceptibility. For attacks based on
adversarial examples, a second crucial parameter is the absence of local
concentrations in the data probability distribution, a property known as
Smeared Absolute Continuity. According to our findings, robustness to
adversarial examples requires either (a) the data distributions in the AI's
feature space to have concentrated probability density functions or (b) the
dimensionality of the AI's decision variables to be sufficiently small. We also
show how to construct stealth attacks on high-dimensional AI systems that are
hard to spot unless the validation set is made exponentially large.

negative:
Vision Transformer (ViT) and its variants (e.g., Swin, PVT) have achieved
great success in various computer vision tasks, owing to their capability to
learn long-range contextual information. Layer Normalization (LN) is an
essential ingredient in these models. However, we found that the ordinary LN
makes tokens at different positions similar in magnitude because it normalizes
embeddings within each token. It is difficult for Transformers to capture
inductive bias such as the positional context in an image with LN. We tackle
this problem by proposing a new normalizer, termed Dynamic Token Normalization
(DTN), where normalization is performed both within each token (intra-token)
and across different tokens (inter-token). DTN has several merits. Firstly, it
is built on a unified formulation and thus can represent various existing
normalization methods. Secondly, DTN learns to normalize tokens in both
intra-token and inter-token manners, enabling Transformers to capture both the
global contextual information and the local positional context. {Thirdly, by
simply replacing LN layers, DTN can be readily plugged into various vision
transformers, such as ViT, Swin, PVT, LeViT, T2T-ViT, BigBird and Reformer.
Extensive experiments show that the transformer equipped with DTN consistently
outperforms baseline model with minimal extra parameters and computational
overhead. For example, DTN outperforms LN by $0.5\%$ - $1.2\%$ top-1 accuracy
on ImageNet, by $1.2$ - $1.4$ box AP in object detection on COCO benchmark, by
$2.3\%$ - $3.9\%$ mCE in robustness experiments on ImageNet-C, and by $0.5\%$ -
$0.8\%$ accuracy in Long ListOps on Long-Range Arena.} Codes will be made
public at \url{https://github.com/wqshao126/DTN}

negative:
Current conversational AI systems aim to understand a set of pre-designed
requests and execute related actions, which limits them to evolve naturally and
adapt based on human interactions. Motivated by how children learn their first
language interacting with adults, this paper describes a new Teachable AI
system that is capable of learning new language nuggets called concepts,
directly from end users using live interactive teaching sessions. The proposed
setup uses three models to: a) Identify gaps in understanding automatically
during live conversational interactions, b) Learn the respective
interpretations of such unknown concepts from live interactions with users, and
c) Manage a classroom sub-dialogue specifically tailored for interactive
teaching sessions. We propose state-of-the-art transformer based neural
architectures of models, fine-tuned on top of pre-trained models, and show
accuracy improvements on the respective components. We demonstrate that this
method is very promising in leading way to build more adaptive and personalized
language understanding models.

negative:
Conversational agents ("bots") are beginning to be widely used in
conversational interfaces. To design a system that is capable of emulating
human-like interactions, a conversational layer that can serve as a fabric for
chat-like interaction with the agent is needed. In this paper, we introduce a
model that employs Information Retrieval by utilizing convolutional deep
structured semantic neural network-based features in the ranker to present
human-like responses in ongoing conversation with a user. In conversations,
accounting for context is critical to the retrieval model; we show that our
context-sensitive approach using a Convolutional Deep Structured Semantic Model
(cDSSM) with character trigrams significantly outperforms several conventional
baselines in terms of the relevance of responses retrieved.

negative:
Over the last several years, end-to-end neural conversational agents have
vastly improved in their ability to carry a chit-chat conversation with humans.
However, these models are often trained on large datasets from the internet,
and as a result, may learn undesirable behaviors from this data, such as toxic
or otherwise harmful language. Researchers must thus wrestle with the issue of
how and when to release these models. In this paper, we survey the problem
landscape for safety for end-to-end conversational AI and discuss recent and
related work. We highlight tensions between values, potential positive impact
and potential harms, and provide a framework for making decisions about whether
and how to release these models, following the tenets of value-sensitive
design. We additionally provide a suite of tools to enable researchers to make
better-informed decisions about training and releasing end-to-end
conversational AI models.

negative:
Despite the tremendous success of deep neural networks in various learning
problems, it has been observed that adding an intentionally designed
adversarial perturbation to inputs of these architectures leads to erroneous
classification with high confidence in the prediction. In this work, we propose
a general framework based on the perturbation analysis of learning algorithms
which consists of convex programming and is able to recover many current
adversarial attacks as special cases. The framework can be used to propose
novel attacks against learning algorithms for classification and regression
tasks under various new constraints with closed form solutions in many
instances. In particular we derive new attacks against classification
algorithms which are shown to achieve comparable performances to notable
existing attacks. The framework is then used to generate adversarial
perturbations for regression tasks which include single pixel and single subset
attacks. By applying this method to autoencoding and image colorization tasks,
it is shown that adversarial perturbations can effectively perturb the output
of regression tasks as well.

negative:
Context-aware neural machine translation (NMT) is a promising direction to
improve the translation quality by making use of the additional context, e.g.,
document-level translation, or having meta-information. Although there exist
various architectures and analyses, the effectiveness of different
context-aware NMT models is not well explored yet. This paper analyzes the
performance of document-level NMT models on four diverse domains with a varied
amount of parallel document-level bilingual data. We conduct a comprehensive
set of experiments to investigate the impact of document-level NMT. We find
that there is no single best approach to document-level NMT, but rather that
different architectures come out on top on different tasks. Looking at
task-specific problems, such as pronoun resolution or headline translation, we
find improvements in the context-aware systems, even in cases where the
corpus-level metrics like BLEU show no significant improvement. We also show
that document-level back-translation significantly helps to compensate for the
lack of document-level bi-texts.

negative:
AI systems have seen significant adoption in various domains. At the same
time, further adoption in some domains is hindered by inability to fully trust
an AI system that it will not harm a human. Besides the concerns for fairness,
privacy, transparency, and explainability are key to developing trusts in AI
systems. As stated in describing trustworthy AI "Trust comes through
understanding. How AI-led decisions are made and what determining factors were
included are crucial to understand." The subarea of explaining AI systems has
come to be known as XAI. Multiple aspects of an AI system can be explained;
these include biases that the data might have, lack of data points in a
particular region of the example space, fairness of gathering the data, feature
importances, etc. However, besides these, it is critical to have human-centered
explanations that are directly related to decision-making similar to how a
domain expert makes decisions based on "domain knowledge," that also include
well-established, peer-validated explicit guidelines. To understand and
validate an AI system's outcomes (such as classification, recommendations,
predictions), that lead to developing trust in the AI system, it is necessary
to involve explicit domain knowledge that humans understand and use.

negative:
The application of cognitive mechanisms to support knowledge acquisition is,
from our point of view, crucial for making the resulting models coherent,
efficient, credible, easy to use and understandable. In particular, there are
two characteristic features of intelligence that are essential for knowledge
development: forgetting and consolidation. Both plays an important role in
knowledge bases and learning systems to avoid possible information overflow and
redundancy, and in order to preserve and strengthen important or frequently
used rules and remove (or forget) useless ones. We present an incremental,
long-life view of knowledge acquisition which tries to improve task after task
by determining what to keep, what to consolidate and what to forget, overcoming
The Stability-Plasticity dilemma. In order to do that, we rate rules by
introducing several metrics through the first adaptation, to our knowledge, of
the Minimum Message Length (MML) principle to a coverage graph, a hierarchical
assessment structure which treats evidence and rules in a unified way. The
metrics are not only used to forget some of the worst rules, but also to set a
consolidation process to promote those selected rules to the knowledge base,
which is also mirrored by a demotion system. We evaluate the framework with a
series of tasks in a chess rule learning domain.

negative:
eXplainable AI focuses on generating explanations for the output of an AI
algorithm to a user, usually a decision-maker. Such user needs to interpret the
AI system in order to decide whether to trust the machine outcome. When
addressing this challenge, therefore, proper attention should be given to
produce explanations that are interpretable by the target community of users.
In this chapter, we claim for the need to better investigate what constitutes a
human explanation, i.e. a justification of the machine behaviour that is
interpretable and actionable by the human decision makers. In particular, we
focus on the contributions that Human Intelligence can bring to eXplainable AI,
especially in conjunction with the exploitation of Knowledge Graphs. Indeed, we
call for a better interplay between Knowledge Representation and Reasoning,
Social Sciences, Human Computation and Human-Machine Cooperation research -- as
already explored in other AI branches -- in order to support the goal of
eXplainable AI with the adoption of a Human-in-the-Loop approach.

negative:
Owing to the rapidly growing multimedia content available on the Internet,
extractive spoken document summarization, with the purpose of automatically
selecting a set of representative sentences from a spoken document to concisely
express the most important theme of the document, has been an active area of
research and experimentation. On the other hand, word embedding has emerged as
a newly favorite research subject because of its excellent performance in many
natural language processing (NLP)-related tasks. However, as far as we are
aware, there are relatively few studies investigating its use in extractive
text or speech summarization. A common thread of leveraging word embeddings in
the summarization process is to represent the document (or sentence) by
averaging the word embeddings of the words occurring in the document (or
sentence). Then, intuitively, the cosine similarity measure can be employed to
determine the relevance degree between a pair of representations. Beyond the
continued efforts made to improve the representation of words, this paper
focuses on building novel and efficient ranking models based on the general
word embedding methods for extractive speech summarization. Experimental
results demonstrate the effectiveness of our proposed methods, compared to
existing state-of-the-art methods.

negative:
A key challenge of dialog systems research is to effectively and efficiently
adapt to new domains. A scalable paradigm for adaptation necessitates the
development of generalizable models that perform well in few-shot settings. In
this paper, we focus on the intent classification problem which aims to
identify user intents given utterances addressed to the dialog system. We
propose two approaches for improving the generalizability of utterance
classification models: (1) observers and (2) example-driven training. Prior
work has shown that BERT-like models tend to attribute a significant amount of
attention to the [CLS] token, which we hypothesize results in diluted
representations. Observers are tokens that are not attended to, and are an
alternative to the [CLS] token as a semantic representation of utterances.
Example-driven training learns to classify utterances by comparing to examples,
thereby using the underlying encoder as a sentence similarity model. These
methods are complementary; improving the representation through observers
allows the example-driven model to better measure sentence similarities. When
combined, the proposed methods attain state-of-the-art results on three intent
prediction datasets (\textsc{banking77}, \textsc{clinc150}, \textsc{hwu64}) in
both the full data and few-shot (10 examples per intent) settings. Furthermore,
we demonstrate that the proposed approach can transfer to new intents and
across datasets without any additional training.

negative:
Recently we see a rising number of methods in the field of eXplainable
Artificial Intelligence. To our surprise, their development is driven by model
developers rather than a study of needs for human end users. The analysis of
needs, if done, takes the form of an A/B test rather than a study of open
questions. To answer the question "What would a human operator like to ask the
ML model?" we propose a conversational system explaining decisions of the
predictive model. In this experiment, we developed a chatbot called dr_ant to
talk about machine learning model trained to predict survival odds on Titanic.
People can talk with dr_ant about different aspects of the model to understand
the rationale behind its predictions. Having collected a corpus of 1000+
dialogues, we analyse the most common types of questions that users would like
to ask. To our knowledge, it is the first study which uses a conversational
system to collect the needs of human operators from the interactive and
iterative dialogue explorations of a predictive model.

negative:
We describe the setting and results of the ConvAI2 NeurIPS competition that
aims to further the state-of-the-art in open-domain chatbots. Some key
takeaways from the competition are: (i) pretrained Transformer variants are
currently the best performing models on this task, (ii) but to improve
performance on multi-turn conversations with humans, future systems must go
beyond single word metrics like perplexity to measure the performance across
sequences of utterances (conversations) -- in terms of repetition, consistency
and balance of dialogue acts (e.g. how many questions asked vs. answered).

negative:
For the next step in human to machine interaction, Artificial Intelligence
(AI) should interact predominantly using natural language because, if it
worked, it would be the fastest way to communicate. Facebook's toy tasks (bAbI)
provide a useful benchmark to compare implementations for conversational AI.
While the published experiments so far have been based on exploiting the
distributional hypothesis with machine learning, our model exploits natural
language understanding (NLU) with the decomposition of language based on Role
and Reference Grammar (RRG) and the brain-based Patom theory. Our combinatorial
system for conversational AI based on linguistics has many advantages: passing
bAbI task tests without parsing or statistics while increasing scalability. Our
model validates both the training and test data to find 'garbage' input and
output (GIGO). It is not rules-based, nor does it use parts of speech, but
instead relies on meaning. While Deep Learning is difficult to debug and fix,
every step in our model can be understood and changed like any non-statistical
computer program. Deep Learning's lack of explicable reasoning has raised
opposition to AI, partly due to fear of the unknown. To support the goals of
AI, we propose extended tasks to use human-level statements with tense, aspect
and voice, and embedded clauses with junctures: and answers to be natural
language generation (NLG) instead of keywords. While machine learning permits
invalid training data to produce incorrect test responses, our system cannot
because the context tracking would need to be intentionally broken. We believe
no existing learning systems can currently solve these extended natural
language tests. There appears to be a knowledge gap between NLP researchers and
linguists, but ongoing competitive results such as these promise to narrow that
gap.

negative:
Endowing a chatbot with personality or an identity is quite challenging but
critical to deliver more realistic and natural conversations. In this paper, we
address the issue of generating responses that are coherent to a pre-specified
agent profile. We design a model consisting of three modules: a profile
detector to decide whether a post should be responded using the profile and
which key should be addressed, a bidirectional decoder to generate responses
forward and backward starting from a selected profile value, and a position
detector that predicts a word position from which decoding should start given a
selected profile value. We show that general conversation data from social
media can be used to generate profile-coherent responses. Manual and automatic
evaluation shows that our model can deliver more coherent, natural, and
diversified responses.

negative:
Document-level relation extraction aims to discover relations between
entities across a whole document. How to build the dependency of entities from
different sentences in a document remains to be a great challenge. Current
approaches either leverage syntactic trees to construct document-level graphs
or aggregate inference information from different sentences. In this paper, we
build cross-sentence dependencies by inferring compositional relations between
inter-sentence mentions. Adopting aggressive linking strategy, intermediate
relations are reasoned on the document-level graphs by mention convolution. We
further notice the generalization problem of NA instances, which is caused by
incomplete annotation and worsened by fully-connected mention pairs. An
improved ranking loss is proposed to attend this problem. Experiments show the
connections between different mentions are crucial to document-level relation
extraction, which enables the model to extract more meaningful higher-level
compositional relations.

negative:
Artificial intelligence (AI) has huge potential to improve the health and
well-being of people, but adoption in clinical practice is still limited. Lack
of transparency is identified as one of the main barriers to implementation, as
clinicians should be confident the AI system can be trusted. Explainable AI has
the potential to overcome this issue and can be a step towards trustworthy AI.
In this paper we review the recent literature to provide guidance to
researchers and practitioners on the design of explainable AI systems for the
health-care domain and contribute to formalization of the field of explainable
AI. We argue the reason to demand explainability determines what should be
explained as this determines the relative importance of the properties of
explainability (i.e. interpretability and fidelity). Based on this, we propose
a framework to guide the choice between classes of explainable AI methods
(explainable modelling versus post-hoc explanation; model-based,
attribution-based, or example-based explanations; global and local
explanations). Furthermore, we find that quantitative evaluation metrics, which
are important for objective standardized evaluation, are still lacking for some
properties (e.g. clarity) and types of explanations (e.g. example-based
methods). We conclude that explainable modelling can contribute to trustworthy
AI, but the benefits of explainability still need to be proven in practice and
complementary measures might be needed to create trustworthy AI in health care
(e.g. reporting data quality, performing extensive (external) validation, and
regulation).

negative:
AI practitioners typically strive to develop the most accurate systems,
making an implicit assumption that the AI system will function autonomously.
However, in practice, AI systems often are used to provide advice to people in
domains ranging from criminal justice and finance to healthcare. In such
AI-advised decision making, humans and machines form a team, where the human is
responsible for making final decisions. But is the most accurate AI the best
teammate? We argue "No" -- predictable performance may be worth a slight
sacrifice in AI accuracy. Instead, we argue that AI systems should be trained
in a human-centered manner, directly optimized for team performance. We study
this proposal for a specific type of human-AI teaming, where the human overseer
chooses to either accept the AI recommendation or solve the task themselves. To
optimize the team performance for this setting we maximize the team's expected
utility, expressed in terms of the quality of the final decision, cost of
verifying, and individual accuracies of people and machines. Our experiments
with linear and non-linear models on real-world, high-stakes datasets show that
the most accuracy AI may not lead to highest team performance and show the
benefit of modeling teamwork during training through improvements in expected
team utility across datasets, considering parameters such as human skill and
the cost of mistakes. We discuss the shortcoming of current optimization
approaches beyond well-studied loss functions such as log-loss, and encourage
future work on AI optimization problems motivated by human-AI collaboration.

negative:
Most typical click models assume that the probability of a document to be
examined by users only depends on position, such as PBM and UBM. It works well
in various kinds of search engines. However, in a search engine where massive
candidate documents display images as responses to the query, the examination
probability should not only depend on position. The visual appearance of an
image-oriented document also plays an important role in its opportunity to be
examined. In this paper, we assume that vision bias exists in an image-oriented
search engine as another crucial factor affecting the examination probability
aside from position. Specifically, we apply this assumption to classical click
models and propose an extended model, to better capture the examination
probabilities of documents. We use regression-based EM algorithm to predict the
vision bias given the visual features extracted from candidate documents.
Empirically, we evaluate our model on a dataset developed from a real-world
online image-oriented search engine, and demonstrate that our proposed model
can achieve significant improvements over its baseline model in data fitness
and sparsity handling.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
In this paper we discuss the relationships between conditional and
preferential logics and neural network models, based on a multi-preferential
semantics. We propose a concept-wise multipreference semantics, recently
introduced for defeasible description logics to take into account preferences
with respect to different concepts, as a tool for providing a semantic
interpretation to neural network models. This approach has been explored both
for unsupervised neural network models (Self-Organising Maps) and for
supervised ones (Multilayer Perceptrons), and we expect that the same approach
might be extended to other neural network models. It allows for logical
properties of the network to be checked (by model checking) over an
interpretation capturing the input-output behavior of the network. For
Multilayer Perceptrons, the deep network itself can be regarded as a
conditional knowledge base, in which synaptic connections correspond to
weighted conditionals. The paper describes the general approach, through the
cases of Self-Organising Maps and Multilayer Perceptrons, and discusses some
open issues and perspectives.

negative:
We develop and study new adversarial perturbations that enable an attacker to
gain control over decisions in generic Artificial Intelligence (AI) systems
including deep learning neural networks. In contrast to adversarial data
modification, the attack mechanism we consider here involves alterations to the
AI system itself. Such a stealth attack could be conducted by a mischievous,
corrupt or disgruntled member of a software development team. It could also be
made by those wishing to exploit a ``democratization of AI'' agenda, where
network architectures and trained parameter sets are shared publicly. We
develop a range of new implementable attack strategies with accompanying
analysis, showing that with high probability a stealth attack can be made
transparent, in the sense that system performance is unchanged on a fixed
validation set which is unknown to the attacker, while evoking any desired
output on a trigger input of interest. The attacker only needs to have
estimates of the size of the validation set and the spread of the AI's relevant
latent space. In the case of deep learning neural networks, we show that a one
neuron attack is possible - a modification to the weights and bias associated
with a single neuron - revealing a vulnerability arising from
over-parameterization. We illustrate these concepts using state of the art
architectures on two standard image data sets. Guided by the theory and
computational results, we also propose strategies to guard against stealth
attacks.

negative:
Trust is a central component of the interaction between people and AI, in
that 'incorrect' levels of trust may cause misuse, abuse or disuse of the
technology. But what, precisely, is the nature of trust in AI? What are the
prerequisites and goals of the cognitive mechanism of trust, and how can we
promote them, or assess whether they are being satisfied in a given
interaction? This work aims to answer these questions. We discuss a model of
trust inspired by, but not identical to, sociology's interpersonal trust (i.e.,
trust between people). This model rests on two key properties of the
vulnerability of the user and the ability to anticipate the impact of the AI
model's decisions. We incorporate a formalization of 'contractual trust', such
that trust between a user and an AI is trust that some implicit or explicit
contract will hold, and a formalization of 'trustworthiness' (which detaches
from the notion of trustworthiness in sociology), and with it concepts of
'warranted' and 'unwarranted' trust. We then present the possible causes of
warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how
to design trustworthy AI, how to evaluate whether trust has manifested, and
whether it is warranted. Finally, we elucidate the connection between trust and
XAI using our formalization.

negative:
The widespread utilization of AI systems has drawn attention to the potential
impacts of such systems on society. Of particular concern are the consequences
that prediction errors may have on real-world scenarios, and the trust humanity
places in AI systems. It is necessary to understand how we can evaluate
trustworthiness in AI and how individuals and entities alike can develop
trustworthy AI systems. In this paper, we analyze each element of
trustworthiness and provide a set of 20 guidelines that can be leveraged to
ensure optimal AI functionality while taking into account the greater ethical,
technical, and practical impacts to humanity. Moreover, the guidelines help
ensure that trustworthiness is provable and can be demonstrated, they are
implementation agnostic, and they can be applied to any AI system in any
sector.

negative:
The role of conversational assistants has become more prevalent in helping
people increase their productivity. Document-centered assistance, for example
to help an individual quickly review a document, has seen less significant
progress, even though it has the potential to tremendously increase a user's
productivity. This type of document-centered assistance is the focus of this
paper. Our contributions are three-fold: (1) We first present a survey to
understand the space of document-centered assistance and the capabilities
people expect in this scenario. (2) We investigate the types of queries that
users will pose while seeking assistance with documents, and show that
document-centered questions form the majority of these queries. (3) We present
a set of initial machine learned models that show that (a) we can accurately
detect document-centered questions, and (b) we can build reasonably accurate
models for answering such questions. These positive results are encouraging,
and suggest that even greater results may be attained with continued study of
this interesting and novel problem space. Our findings have implications for
the design of intelligent systems to support task completion via natural
interactions with documents.

negative:
People have information needs of varying complexity, which can be solved by
an intelligent agent able to answer questions formulated in a proper way,
eventually considering user context and preferences. In a scenario in which the
user profile can be considered as a question, intelligent agents able to answer
questions can be used to find the most relevant answers for a given user. In
this work we propose a novel model based on Artificial Neural Networks to
answer questions with multiple answers by exploiting multiple facts retrieved
from a knowledge base. The model is evaluated on the factoid Question Answering
and top-n recommendation tasks of the bAbI Movie Dialog dataset. After
assessing the performance of the model on both tasks, we try to define the
long-term goal of a conversational recommender system able to interact using
natural language and to support users in their information seeking processes in
a personalized way.

negative:
Building a dialogue system that can communicate naturally with humans is a
challenging yet interesting problem of agent-based computing. The rapid growth
in this area is usually hindered by the long-standing problem of data scarcity
as these systems are expected to learn syntax, grammar, decision making, and
reasoning from insufficient amounts of task-specific dataset. The recently
introduced pre-trained language models have the potential to address the issue
of data scarcity and bring considerable advantages by generating contextualized
word embeddings. These models are considered counterpart of ImageNet in NLP and
have demonstrated to capture different facets of language such as hierarchical
relations, long-term dependency, and sentiment. In this short survey paper, we
discuss the recent progress made in the field of pre-trained language models.
We also deliberate that how the strengths of these language models can be
leveraged in designing more engaging and more eloquent conversational agents.
This paper, therefore, intends to establish whether these pre-trained models
can overcome the challenges pertinent to dialogue systems, and how their
architecture could be exploited in order to overcome these challenges. Open
challenges in the field of dialogue systems have also been deliberated.

negative:
Today is the era of intelligence in machines. With the advances in Artificial
Intelligence, machines have started to impersonate different human traits, a
chatbot is the next big thing in the domain of conversational services. A
chatbot is a virtual person who is capable to carry out a natural conversation
with people. They can include skills that enable them to converse with the
humans in audio, visual, or textual formats. Artificial intelligence
conversational entities, also called chatbots, conversational agents, or
dialogue system, are an excellent example of such machines. Obtaining the right
information at the right time and place is the key to effective disaster
management. The term "disaster management" encompasses both natural and
human-caused disasters. To assist citizens, our project is to create a COVID
Assistant to provide the need of up to date information to be available 24
hours. With the growth in the World Wide Web, it is quite intelligible that
users are interested in the swift and relatedly correct information for their
hunt. A chatbot can be seen as a question-and-answer system in which experts
provide knowledge to solicit users. This master thesis is dedicated to discuss
COVID Assistant chatbot and explain each component in detail. The design of the
proposed chatbot is introduced by its seven components: Ontology, Web Scraping
module, DB, State Machine, keyword Extractor, Trained chatbot, and User
Interface.

negative:
A market-maker-based prediction market lets forecasters aggregate information
by editing a consensus probability distribution either directly or by trading
securities that pay off contingent on an event of interest. Combinatorial
prediction markets allow trading on any event that can be specified as a
combination of a base set of events. However, explicitly representing the full
joint distribution is infeasible for markets with more than a few base events.
A factored representation such as a Bayesian network (BN) can achieve tractable
computation for problems with many related variables. Standard BN inference
algorithms, such as the junction tree algorithm, can be used to update a
representation of the entire joint distribution given a change to any local
conditional probability. However, in order to let traders reuse assets from
prior trades while never allowing assets to become negative, a BN based
prediction market also needs to update a representation of each user's assets
and find the conditional state in which a user has minimum assets. Users also
find it useful to see their expected assets given an edit outcome. We show how
to generalize the junction tree algorithm to perform all these computations.

negative:
In the current era, people and society have grown increasingly reliant on
artificial intelligence (AI) technologies. AI has the potential to drive us
towards a future in which all of humanity flourishes. It also comes with
substantial risks for oppression and calamity. Discussions about whether we
should (re)trust AI have repeatedly emerged in recent years and in many
quarters, including industry, academia, healthcare, services, and so on.
Technologists and AI researchers have a responsibility to develop trustworthy
AI systems. They have responded with great effort to design more responsible AI
algorithms. However, existing technical solutions are narrow in scope and have
been primarily directed towards algorithms for scoring or classification tasks,
with an emphasis on fairness and unwanted bias. To build long-lasting trust
between AI and human beings, we argue that the key is to think beyond
algorithmic fairness and connect major aspects of AI that potentially cause
AI's indifferent behavior. In this survey, we provide a systematic framework of
Socially Responsible AI Algorithms that aims to examine the subjects of AI
indifference and the need for socially responsible AI algorithms, define the
objectives, and introduce the means by which we may achieve these objectives.
We further discuss how to leverage this framework to improve societal
well-being through protection, information, and prevention/mitigation.

negative:
Many name tagging approaches use local contextual information with much
success, but fail when the local context is ambiguous or limited. We present a
new framework to improve name tagging by utilizing local, document-level, and
corpus-level contextual information. We retrieve document-level context from
other sentences within the same document and corpus-level context from
sentences in other topically related documents. We propose a model that learns
to incorporate document-level and corpus-level contextual information alongside
local contextual information via global attentions, which dynamically weight
their respective contextual information, and gating mechanisms, which determine
the influence of this information. Extensive experiments on benchmark datasets
show the effectiveness of our approach, which achieves state-of-the-art results
for Dutch, German, and Spanish on the CoNLL-2002 and CoNLL-2003 datasets.

negative:
Recent advances in artificial intelligence (AI) and machine learning have
created a general perception that AI could be used to solve complex problems,
and in some situations over-hyped as a tool that can be so easily used.
Unfortunately, the barrier to realization of mass adoption of AI on various
business domains is too high because most domain experts have no background in
AI. Developing AI applications involves multiple phases, namely data
preparation, application modeling, and product deployment. The effort of AI
research has been spent mostly on new AI models (in the model training stage)
to improve the performance of benchmark tasks such as image recognition. Many
other factors such as usability, efficiency and security of AI have not been
well addressed, and therefore form a barrier to democratizing AI. Further, for
many real world applications such as healthcare and autonomous driving,
learning via huge amounts of possibility exploration is not feasible since
humans are involved. In many complex applications such as healthcare, subject
matter experts (e.g. Clinicians) are the ones who appreciate the importance of
features that affect health, and their knowledge together with existing
knowledge bases are critical to the end results. In this paper, we take a new
perspective on developing AI solutions, and present a solution for making AI
usable. We hope that this resolution will enable all subject matter experts
(eg. Clinicians) to exploit AI like data scientists.

negative:
Chatbot is a technology that is used to mimic human behavior using natural
language. There are different types of Chatbot that can be used as
conversational agent in various business domains in order to increase the
customer service and satisfaction. For any business domain, it requires a
knowledge base to be built for that domain and design an information retrieval
based system that can respond the user with a piece of documentation or
generated sentences. The core component of a Chatbot is Natural Language
Understanding (NLU) which has been impressively improved by deep learning
methods. But we often lack such properly built NLU modules and requires more
time to build it from scratch for high quality conversations. This may
encourage fresh learners to build a Chatbot from scratch with simple
architecture and using small dataset, although it may have reduced
functionality, rather than building high quality data driven methods. This
research focuses on Named Entity Recognition (NER) and Intent Classification
models which can be integrated into NLU service of a Chatbot. Named entities
will be inserted manually in the knowledge base and automatically detected in a
given sentence. The NER model in the proposed architecture is based on
artificial neural network which is trained on manually created entities and
evaluated using CoNLL-2003 dataset.

negative:
Today, AI is being increasingly used to help human experts make decisions in
high-stakes scenarios. In these scenarios, full automation is often
undesirable, not only due to the significance of the outcome, but also because
human experts can draw on their domain knowledge complementary to the model's
to ensure task success. We refer to these scenarios as AI-assisted decision
making, where the individual strengths of the human and the AI come together to
optimize the joint decision outcome. A key to their success is to appropriately
\textit{calibrate} human trust in the AI on a case-by-case basis; knowing when
to trust or distrust the AI allows the human expert to appropriately apply
their knowledge, improving decision outcomes in cases where the model is likely
to perform poorly. This research conducts a case study of AI-assisted decision
making in which humans and AI have comparable performance alone, and explores
whether features that reveal case-specific model information can calibrate
trust and improve the joint performance of the human and AI. Specifically, we
study the effect of showing confidence score and local explanation for a
particular prediction. Through two human experiments, we show that confidence
score can help calibrate people's trust in an AI model, but trust calibration
alone is not sufficient to improve AI-assisted decision making, which may also
depend on whether the human can bring in enough unique knowledge to complement
the AI's errors. We also highlight the problems in using local explanation for
AI-assisted decision making scenarios and invite the research community to
explore new approaches to explainability for calibrating human trust in AI.

negative:
With the complexity of the network structure, uncertainty inference has
become an important task to improve the classification accuracy for artificial
intelligence systems. For image classification tasks, we propose a structured
DropConnect (SDC) framework to model the output of a deep neural network by a
Dirichlet distribution. We introduce a DropConnect strategy on weights in the
fully connected layers during training. In test, we split the network into
several sub-networks, and then model the Dirichlet distribution by match its
moments with the mean and variance of the outputs of these sub-networks. The
entropy of the estimated Dirichlet distribution is finally utilized for
uncertainty inference. In this paper, this framework is implemented on LeNet$5$
and VGG$16$ models for misclassification detection and out-of-distribution
detection on MNIST and CIFAR-$10$ datasets. Experimental results show that the
performance of the proposed SDC can be comparable to other uncertainty
inference methods. Furthermore, the SDC is adapted well to different network
structures with certain generalization capabilities and research prospects.

negative:
In adversarial training, a set of models learn together by pursuing competing
goals, usually defined on single data instances. However, in relational
learning and other non-i.i.d domains, goals can also be defined over sets of
instances. For example, a link predictor for the is-a relation needs to be
consistent with the transitivity property: if is-a(x_1, x_2) and is-a(x_2, x_3)
hold, is-a(x_1, x_3) needs to hold as well. Here we use such assumptions for
deriving an inconsistency loss, measuring the degree to which the model
violates the assumptions on an adversarially-generated set of examples. The
training objective is defined as a minimax problem, where an adversary finds
the most offending adversarial examples by maximising the inconsistency loss,
and the model is trained by jointly minimising a supervised loss and the
inconsistency loss on the adversarial examples. This yields the first method
that can use function-free Horn clauses (as in Datalog) to regularise any
neural link predictor, with complexity independent of the domain size. We show
that for several link prediction models, the optimisation problem faced by the
adversary has efficient closed-form solutions. Experiments on link prediction
benchmarks indicate that given suitable prior knowledge, our method can
significantly improve neural link predictors on all relevant metrics.

negative:
Neural networks with deep architectures have demonstrated significant
performance improvements in computer vision, speech recognition, and natural
language processing. The challenges in information retrieval (IR), however, are
different from these other application areas. A common form of IR involves
ranking of documents--or short passages--in response to keyword-based queries.
Effective IR systems must deal with query-document vocabulary mismatch problem,
by modeling relationships between different query and document terms and how
they indicate relevance. Models should also consider lexical matches when the
query contains rare terms--such as a person's name or a product model
number--not seen during training, and to avoid retrieving semantically related
but irrelevant results. In many real-life IR tasks, the retrieval involves
extremely large collections--such as the document index of a commercial Web
search engine--containing billions of documents. Efficient IR methods should
take advantage of specialized IR data structures, such as inverted index, to
efficiently retrieve from large collections. Given an information need, the IR
system also mediates how much exposure an information artifact receives by
deciding whether it should be displayed, and where it should be positioned,
among other results. Exposure-aware IR systems may optimize for additional
objectives, besides relevance, such as parity of exposure for retrieved items
and content publishers. In this thesis, we present novel neural architectures
and methods motivated by the specific needs and challenges of IR tasks.

negative:
Document clustering is a text mining technique used to provide better
document search and browsing in digital libraries or online corpora. A lot of
research has been done on biomedical document clustering that is based on using
existing ontology. But, associations and co-occurrences of the medical concepts
are not well represented by using ontology. In this research, a vector
representation of concepts of diseases and similarity measurement between
concepts are proposed. They identify the closest concepts of diseases in the
context of a corpus. Each document is represented by using the vector space
model. A weight scheme is proposed to consider both local content and
associations between concepts. A Self-Organizing Map is used as document
clustering algorithm. The vector projection and visualization features of SOM
enable visualization and analysis of the clusters distributions and
relationships on the two dimensional space. The experimental results show that
the proposed document clustering framework generates meaningful clusters and
facilitate visualization of the clusters based on the concepts of diseases.

negative:
Black-box machine learning models are now routinely used in high-risk
settings, like medical diagnostics, which demand uncertainty quantification to
avoid consequential model failures. Conformal prediction is a user-friendly
paradigm for creating statistically rigorous uncertainty sets/intervals for the
predictions of such models. Critically, the sets are valid in a
distribution-free sense: they possess explicit, non-asymptotic guarantees even
without distributional assumptions or model assumptions. One can use conformal
prediction with any pre-trained model, such as a neural network, to produce
sets that are guaranteed to contain the ground truth with a user-specified
probability, such as 90%. It is easy-to-understand, easy-to-use, and general,
applying naturally to problems arising in the fields of computer vision,
natural language processing, deep reinforcement learning, and so on.
  This hands-on introduction is aimed to provide the reader a working
understanding of conformal prediction and related distribution-free uncertainty
quantification techniques with one self-contained document. We lead the reader
through practical theory for and examples of conformal prediction and describe
its extensions to complex machine learning tasks involving structured outputs,
distribution shift, time-series, outliers, models that abstain, and more.
Throughout, there are many explanatory illustrations, examples, and code
samples in Python. With each code sample comes a Jupyter notebook implementing
the method on a real-data example; the notebooks can be accessed and easily run
using our codebase.

negative:
Over the past few years, providers such as Google, Microsoft, and Amazon have
started to provide customers with access to software interfaces allowing them
to easily embed machine learning tasks into their applications. Overall,
organizations can now use Machine Learning as a Service (MLaaS) engines to
outsource complex tasks, e.g., training classifiers, performing predictions,
clustering, etc. They can also let others query models trained on their data.
Naturally, this approach can also be used (and is often advocated) in other
contexts, including government collaborations, citizen science projects, and
business-to-business partnerships. However, if malicious users were able to
recover data used to train these models, the resulting information leakage
would create serious issues. Likewise, if the inner parameters of the model are
considered proprietary information, then access to the model should not allow
an adversary to learn such parameters. In this document, we set to review
privacy challenges in this space, providing a systematic review of the relevant
research literature, also exploring possible countermeasures. More
specifically, we provide ample background information on relevant concepts
around machine learning and privacy. Then, we discuss possible adversarial
models and settings, cover a wide range of attacks that relate to private
and/or sensitive information leakage, and review recent results attempting to
defend against such attacks. Finally, we conclude with a list of open problems
that require more work, including the need for better evaluations, more
targeted defenses, and the study of the relation to policy and data protection
efforts.

negative:
Chatbots and AI assistants have claimed their importance in today life. The
main reason behind adopting this technology is to connect with the user,
understand their requirements, and fulfill them. This has been achieved but at
the cost of heavy training data and complex learning models. This work is
carried out proposes a simple algorithm, a model which can be implemented in
different fields each with its own work scope. The proposed model converts
human language text to computer-understandable SQL queries. The model requires
data only related to the specific field, saving data space. This model performs
linear computation hence solving the computational complexity. This work also
defines the stages where a new methodology is implemented and what previous
method was adopted to fulfill the requirement at that stage. Two datasets
available online will be used in this work, the ATIS dataset, and WikiSQL. This
work compares the computation time among the 2 datasets and also compares the
accuracy of both. This paper works over basic Natural language processing tasks
like semantic parsing, NER, parts of speech and tends to achieve results
through these simple methods.

negative:
Instances of Artificial Intelligence (AI) systems failing to deliver
consistent, satisfactory performance are legion. We investigate why AI failures
occur. We address only a narrow subset of the broader field of AI Safety. We
focus on AI failures on account of flaws in conceptualization, design and
deployment. Other AI Safety issues like trade-offs between privacy and security
or convenience, bad actors hacking into AI systems to create mayhem or bad
actors deploying AI for purposes harmful to humanity and are out of scope of
our discussion. We find that AI systems fail on account of omission and
commission errors in the design of the AI system, as well as upon failure to
develop an appropriate interpretation of input information. Moreover, even when
there is no significant flaw in the AI software, an AI system may fail because
the hardware is incapable of robust performance across environments. Finally an
AI system is quite likely to fail in situations where, in effect, it is called
upon to deliver moral judgments -- a capability AI does not possess. We observe
certain trade-offs in measures to mitigate a subset of AI failures and provide
some recommendations.

negative:
The task of next POI recommendation has been studied extensively in recent
years. However, developing an unified recommendation framework to incorporate
multiple factors associated with both POIs and users remains challenging,
because of the heterogeneity nature of these information. Further, effective
mechanisms to handle cold-start and endow the system with interpretability are
also difficult topics. Inspired by the recent success of neural networks in
many areas, in this paper, we present a simple but effective neural network
framework for next POI recommendation, named NEXT. NEXT is an unified framework
to learn the hidden intent regarding user's next move, by incorporating
different factors in an unified manner. Specifically, in NEXT, we incorporate
meta-data information and two kinds of temporal contexts (i.e., time interval
and visit time). To leverage sequential relations and geographical influence,
we propose to adopt DeepWalk, a network representation learning technique, to
encode such knowledge. We evaluate the effectiveness of NEXT against
state-of-the-art alternatives and neural networks based solutions. Experimental
results over three publicly available datasets demonstrate that NEXT
significantly outperforms baselines in real-time next POI recommendation.
Further experiments demonstrate the superiority of NEXT in handling cold-start.
More importantly, we show that NEXT provides meaningful explanation of the
dimensions in hidden intent space.

negative:
Perhaps the most ambitious scientific quest in human history is the creation
of general artificial intelligence, which roughly means AI that is as smart or
smarter than humans. The dominant approach in the machine learning community is
to attempt to discover each of the pieces required for intelligence, with the
implicit assumption that some future group will complete the Herculean task of
figuring out how to combine all of those pieces into a complex thinking
machine. I call this the "manual AI approach". This paper describes another
exciting path that ultimately may be more successful at producing general AI.
It is based on the clear trend in machine learning that hand-designed solutions
eventually are replaced by more effective, learned solutions. The idea is to
create an AI-generating algorithm (AI-GA), which automatically learns how to
produce general AI. Three Pillars are essential for the approach: (1)
meta-learning architectures, (2) meta-learning the learning algorithms
themselves, and (3) generating effective learning environments. I argue that
either approach could produce general AI first, and both are scientifically
worthwhile irrespective of which is the fastest path. Because both are
promising, yet the ML community is currently committed to the manual approach,
I argue that our community should increase its research investment in the AI-GA
approach. To encourage such research, I describe promising work in each of the
Three Pillars. I also discuss AI-GA-specific safety and ethical considerations.
Because it it may be the fastest path to general AI and because it is
inherently scientifically interesting to understand the conditions in which a
simple algorithm can produce general AI (as happened on Earth where Darwinian
evolution produced human intelligence), I argue that the pursuit of AI-GAs
should be considered a new grand challenge of computer science research.

negative:
In recent years, traditional cybersecurity safeguards have proven ineffective
against insider threats. Famous cases of sensitive information leaks caused by
insiders, including the WikiLeaks release of diplomatic cables and the Edward
Snowden incident, have greatly harmed the U.S. government's relationship with
other governments and with its own citizens. Data Leak Prevention (DLP) is a
solution for detecting and preventing information leaks from within an
organization's network. However, state-of-art DLP detection models are only
able to detect very limited types of sensitive information, and research in the
field has been hindered due to the lack of available sensitive texts. Many
researchers have focused on document-based detection with artificially labeled
"confidential documents" for which security labels are assigned to the entire
document, when in reality only a portion of the document is sensitive. This
type of whole-document based security labeling increases the chances of
preventing authorized users from accessing non-sensitive information within
sensitive documents. In this paper, we introduce Automated Classification
Enabled by Security Similarity (ACESS), a new and innovative detection model
that penetrates the complexity of big text security classification/detection.
To analyze the ACESS system, we constructed a novel dataset, containing
formerly classified paragraphs from diplomatic cables made public by the
WikiLeaks organization. To our knowledge this paper is the first to analyze a
dataset that contains actual formerly sensitive information annotated at
paragraph granularity.

negative:
Artificial Neural Networks have reached Grandmaster and even super-human
performance across a variety of games: from those involving perfect-information
(such as Go) to those involving imperfect-information (such as Starcraft). Such
technological developments from AI-labs have ushered concomitant applications
across the world of business - where an AI brand tag is fast becoming
ubiquitous. A corollary of such widespread commercial deployment is that when
AI gets things wrong - an autonomous vehicle crashes; a chatbot exhibits racist
behaviour; automated credit scoring processes discriminate on gender etc. -
there are often significant financial, legal and brand consequences and the
incident becomes major news. As Judea Pearl sees it, the underlying reason for
such mistakes is that, 'all the impressive achievements of deep learning amount
to just curve fitting'. The key, Judea Pearl suggests, is to replace reasoning
by association with causal-reasoning - the ability to infer causes from
observed phenomena. It is a point that was echoed by Gary Marcus and Ernest
Davis in a recent piece for the New York Times: 'we need to stop building
computer systems that merely get better and better at detecting statistical
patterns in data sets - often using an approach known as Deep Learning - and
start building computer systems that from the moment of their assembly innately
grasp three basic concepts: time, space and causality'. In this paper,
foregrounding what in 1949 Gilbert Ryle termed a category mistake, I will offer
an alternative explanation for AI errors: it is not so much that AI machinery
cannot grasp causality, but that AI machinery - qua computation - cannot
understand anything at all.

negative:
AI has provided us with the ability to automate tasks, extract information
from vast amounts of data, and synthesize media that is nearly
indistinguishable from the real thing. However, positive tools can also be used
for negative purposes. In particular, cyber adversaries can use AI (such as
machine learning) to enhance their attacks and expand their campaigns.
  Although offensive AI has been discussed in the past, there is a need to
analyze and understand the threat in the context of organizations. For example,
how does an AI-capable adversary impact the cyber kill chain? Does AI benefit
the attacker more than the defender? What are the most significant AI threats
facing organizations today and what will be their impact on the future?
  In this survey, we explore the threat of offensive AI on organizations.
First, we present the background and discuss how AI changes the adversary's
methods, strategies, goals, and overall attack model. Then, through a
literature review, we identify 33 offensive AI capabilities which adversaries
can use to enhance their attacks. Finally, through a user study spanning
industry and academia, we rank the AI threats and provide insights on the
adversaries.

negative:
This work provides a starting point for researchers interested in gaining a
deeper understanding of the big picture of artificial intelligence (AI). To
this end, a narrative is conveyed that allows the reader to develop an
objective view on current developments that is free from false promises that
dominate public communication. An essential takeaway for the reader is that AI
must be understood as an umbrella term encompassing a plethora of different
methods, schools of thought, and their respective historical movements.
Consequently, a bottom-up strategy is pursued in which the field of AI is
introduced by presenting various aspects that are characteristic of the
subject. This paper is structured in three parts: (i) Discussion of current
trends revealing false public narratives, (ii) an introduction to the history
of AI focusing on recurring patterns and main characteristics, and (iii) a
critical discussion on the limitations of current methods in the context of the
potential emergence of a strong(er) AI. It should be noted that this work does
not cover any of these aspects holistically; rather, the content addressed is a
selection made by the author and subject to a didactic strategy.

negative:
Recently, many AI researchers and practitioners have embarked on research
visions that involve doing AI for "Good". This is part of a general drive
towards infusing AI research and practice with ethical thinking. One frequent
theme in current ethical guidelines is the requirement that AI be good for all,
or: contribute to the Common Good. But what is the Common Good, and is it
enough to want to be good? Via four lead questions, I will illustrate
challenges and pitfalls when determining, from an AI point of view, what the
Common Good is and how it can be enhanced by AI. The questions are: What is the
problem / What is a problem?, Who defines the problem?, What is the role of
knowledge?, and What are important side effects and dynamics? The illustration
will use an example from the domain of "AI for Social Good", more specifically
"Data Science for Social Good". Even if the importance of these questions may
be known at an abstract level, they do not get asked sufficiently in practice,
as shown by an exploratory study of 99 contributions to recent conferences in
the field. Turning these challenges and pitfalls into a positive
recommendation, as a conclusion I will draw on another characteristic of
computer-science thinking and practice to make these impediments visible and
attenuate them: "attacks" as a method for improving design. This results in the
proposal of ethics pen-testing as a method for helping AI designs to better
contribute to the Common Good.

negative:
AI researchers employ not only the scientific method, but also methodology
from mathematics and engineering. However, the use of the scientific method -
specifically hypothesis testing - in AI is typically conducted in service of
engineering objectives. Growing interest in topics such as fairness and
algorithmic bias show that engineering-focused questions only comprise a subset
of the important questions about AI systems. This results in the AI Knowledge
Gap: the number of unique AI systems grows faster than the number of studies
that characterize these systems' behavior. To close this gap, we argue that the
study of AI could benefit from the greater inclusion of researchers who are
well positioned to formulate and test hypotheses about the behavior of AI
systems. We examine the barriers preventing social and behavioral scientists
from conducting such studies. Our diagnosis suggests that accelerating the
scientific study of AI systems requires new incentives for academia and
industry, mediated by new tools and institutions. To address these needs, we
propose a two-sided marketplace called TuringBox. On one side, AI contributors
upload existing and novel algorithms to be studied scientifically by others. On
the other side, AI examiners develop and post machine intelligence tasks
designed to evaluate and characterize algorithmic behavior. We discuss this
market's potential to democratize the scientific study of AI behavior, and thus
narrow the AI Knowledge Gap.

negative:
In this paper I argue that the search for explainable models and
interpretable decisions in AI must be reformulated in terms of the broader
project of offering a pragmatic and naturalistic account of understanding in
AI. Intuitively, the purpose of providing an explanation of a model or a
decision is to make it understandable to its stakeholders. But without a
previous grasp of what it means to say that an agent understands a model or a
decision, the explanatory strategies will lack a well-defined goal. Aside from
providing a clearer objective for XAI, focusing on understanding also allows us
to relax the factivity condition on explanation, which is impossible to fulfill
in many machine learning models, and to focus instead on the pragmatic
conditions that determine the best fit between a model and the methods and
devices deployed to understand it. After an examination of the different types
of understanding discussed in the philosophical and psychological literature, I
conclude that interpretative or approximation models not only provide the best
way to achieve the objectual understanding of a machine learning model, but are
also a necessary condition to achieve post-hoc interpretability. This
conclusion is partly based on the shortcomings of the purely functionalist
approach to post-hoc interpretability that seems to be predominant in most
recent literature.

negative:
Probability theory, epistemically interpreted, provides an excellent, if not
the best available account of inductive reasoning. This is so because there are
general and definite rules for the change of subjective probabilities through
information or experience; induction and belief change are one and same topic,
after all. The most basic of these rules is simply to conditionalize with
respect to the information received; and there are similar and more general
rules. 1 Hence, a fundamental reason for the epistemological success of
probability theory is that there at all exists a well-behaved concept of
conditional probability. Still, people have, and have reasons for, various
concerns over probability theory. One of these is my starting point:
Intuitively, we have the notion of plain belief; we believe propositions2 to be
true (or to be false or neither). Probability theory, however, offers no formal
counterpart to this notion. Believing A is not the same as having probability 1
for A, because probability 1 is incorrigible3; but plain belief is clearly
corrigible. And believing A is not the same as giving A a probability larger
than some 1 - c, because believing A and believing B is usually taken to be
equivalent to believing A & B.4 Thus, it seems that the formal representation
of plain belief has to take a non-probabilistic route. Indeed, representing
plain belief seems easy enough: simply represent an epistemic state by the set
of all propositions believed true in it or, since I make the common assumption
that plain belief is deductively closed, by the conjunction of all propositions
believed true in it. But this does not yet provide a theory of induction, i.e.
an answer to the question how epistemic states so represented are changed
tbrough information or experience. There is a convincing partial answer: if the
new information is compatible with the old epistemic state, then the new
epistemic state is simply represented by the conjunction of the new information
and the old beliefs. This answer is partial because it does not cover the quite
common case where the new information is incompatible with the old beliefs. It
is, however, important to complete the answer and to cover this case, too;
otherwise, we would not represent plain belief as conigible. The crucial
problem is that there is no good completion. When epistemic states are
represented simply by the conjunction of all propositions believed true in it,
the answer cannot be completed; and though there is a lot of fruitful work, no
other representation of epistemic states has been proposed, as far as I know,
which provides a complete solution to this problem. In this paper, I want to
suggest such a solution. In [4], I have more fully argued that this is the only
solution, if certain plausible desiderata are to be satisfied. Here, in section
2, I will be content with formally defining and intuitively explaining my
proposal. I will compare my proposal with probability theory in section 3. It
will turn out that the theory I am proposing is structurally homomorphic to
probability theory in important respects and that it is thus equally easily
implementable, but moreover computationally simpler. Section 4 contains a very
brief comparison with various kinds of logics, in particular conditional logic,
with Shackle's functions of potential surprise and related theories, and with
the Dempster - Shafer theory of belief functions.

negative:
Artificial Intelligence represents many things: a new market to conquer or a
quality label for tech companies, a threat for traditional industries, a menace
for democracy, or a blessing for our busy everyday life. The press abounds in
examples illustrating these aspects, but one should draw not hasty and
premature conclusions. The first successes in AI have been a surprise for
society at large-including researchers in the field. Today, after the initial
stupefaction, we have examples of the system reactions: traditional companies
are heavily investing in AI, social platforms are monitored during elections,
data collection is more and more regulated, etc. The resilience of an
organization (i.e. its capacity to resist to a shock) relies deeply on the
perception of its environment. Future problems have to be anticipated, while
unforeseen events occurring have to be quickly identified in order to be
mitigated as fast as possible. The author states that this clear perception
starts with a common definition of AI in terms of capacities and limits. AI
practitioners should make notions and concepts accessible to the general public
and the impacted fields (e.g. industries, law, education). It is a truism that
only law experts would have the potential to estimate IA impacts on judicial
system. However, questions remain on how to connect different kind of expertise
and what is the appropriate level of detail required for the knowledge
exchanges. And the same consideration is true for dissemination towards
society. Ultimately, society will live with decisions made by the "experts". It
sounds wise to involve society in the decision process rather than risking to
pay consequences later. Therefore, society also needs the key concepts to
understand AI impact on their life. This was the purpose of the trial of an IA
that took place in October 2018 at the Court of Appeal of Paris: gathering
experts from various fields to expose challenges in law and science towards a
general public.

negative:
The range of application of artificial intelligence (AI) is vast, as is the
potential for harm. Growing awareness of potential risks from AI systems has
spurred action to address those risks, while eroding confidence in AI systems
and the organizations that develop them. A 2019 study found over 80
organizations that published and adopted "AI ethics principles'', and more have
joined since. But the principles often leave a gap between the "what" and the
"how" of trustworthy AI development. Such gaps have enabled questionable or
ethically dubious behavior, which casts doubts on the trustworthiness of
specific organizations, and the field more broadly. There is thus an urgent
need for concrete methods that both enable AI developers to prevent harm and
allow them to demonstrate their trustworthiness through verifiable behavior.
Below, we explore mechanisms (drawn from arXiv:2004.07213) for creating an
ecosystem where AI developers can earn trust - if they are trustworthy. Better
assessment of developer trustworthiness could inform user choice, employee
actions, investment decisions, legal recourse, and emerging governance regimes.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.