query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : Deep learning models are notoriously challenging to interpret due to their inherent black-box nature. In my project, I aim to demystify these models by developing alternative, more transparent models that maintain high performance. My current approach involves simplifying the network and reducing its parameters as much as possible without compromising performance. Once the network is sufficiently simplified, we can break it down into more interpretable components. Each component should be simple enough to visualize, and by visualizing the interaction of each component, we can gain a clearer understanding of the entire mechanism and decision process of the black-box model. Consequently, I plan to develop a visualization toolkit to illustrate each component of the simplified neural network model. I will also apply my proposed methodology and toolkit to various real-world scenarios where transparent decision-making is crucial, such as in credit score assessment and loan approval processes.

positive:
Deep neural network (DNN) models have achieved phenomenal success for
applications in many domains, ranging from academic research in science and
engineering to industry and business. The modeling power of DNN is believed to
have come from the complexity and over-parameterization of the model, which on
the other hand has been criticized for the lack of interpretation. Although
certainly not true for every application, in some applications, especially in
economics, social science, healthcare industry, and administrative decision
making, scientists or practitioners are resistant to use predictions made by a
black-box system for multiple reasons. One reason is that a major purpose of a
study can be to make discoveries based upon the prediction function, e.g., to
reveal the relationships between measurements. Another reason can be that the
training dataset is not large enough to make researchers feel completely sure
about a purely data-driven result. Being able to examine and interpret the
prediction function will enable researchers to connect the result with existing
knowledge or gain insights about new directions to explore. Although classic
statistical models are much more explainable, their accuracy often falls
considerably below DNN. In this paper, we propose an approach to fill the gap
between relatively simple explainable models and DNN such that we can more
flexibly tune the trade-off between interpretability and accuracy. Our main
idea is a mixture of discriminative models that is trained with the guidance
from a DNN. Although mixtures of discriminative models have been studied
before, our way of generating the mixture is quite different.

positive:
Deep learning has been shown to outperform traditional machine learning
algorithms across a wide range of problem domains. However, current deep
learning algorithms have been criticized as uninterpretable "black-boxes" which
cannot explain their decision making processes. This is a major shortcoming
that prevents the widespread application of deep learning to domains with
regulatory processes such as finance. As such, industries such as finance have
to rely on traditional models like decision trees that are much more
interpretable but less effective than deep learning for complex problems. In
this paper, we propose CLEAR-Trade, a novel financial AI visualization
framework for deep learning-driven stock market prediction that mitigates the
interpretability issue of deep learning methods. In particular, CLEAR-Trade
provides a effective way to visualize and explain decisions made by deep stock
market prediction models. We show the efficacy of CLEAR-Trade in enhancing the
interpretability of stock market prediction by conducting experiments based on
S&P 500 stock index prediction. The results demonstrate that CLEAR-Trade can
provide significant insight into the decision-making process of deep
learning-driven financial models, particularly for regulatory processes, thus
improving their potential uptake in the financial industry.

positive:
Deep neural network (DNN) models have recently obtained state-of-the-art
prediction accuracy for the transcription factor binding (TFBS) site
classification task. However, it remains unclear how these approaches identify
meaningful DNA sequence signals and give insights as to why TFs bind to certain
locations. In this paper, we propose a toolkit called the Deep Motif Dashboard
(DeMo Dashboard) which provides a suite of visualization strategies to extract
motifs, or sequence patterns from deep neural network models for TFBS
classification. We demonstrate how to visualize and understand three important
DNN models: convolutional, recurrent, and convolutional-recurrent networks. Our
first visualization method is finding a test sequence's saliency map which uses
first-order derivatives to describe the importance of each nucleotide in making
the final prediction. Second, considering recurrent models make predictions in
a temporal manner (from one end of a TFBS sequence to the other), we introduce
temporal output scores, indicating the prediction score of a model over time
for a sequential input. Lastly, a class-specific visualization strategy finds
the optimal input sequence for a given TFBS positive class via stochastic
gradient optimization. Our experimental results indicate that a
convolutional-recurrent architecture performs the best among the three
architectures. The visualization techniques indicate that CNN-RNN makes
predictions by modeling both motifs as well as dependencies among them.

positive:
For reliability, it is important that the predictions made by machine
learning methods are interpretable by human. In general, deep neural networks
(DNNs) can provide accurate predictions, although it is difficult to interpret
why such predictions are obtained by DNNs. On the other hand, interpretation of
linear models is easy, although their predictive performance would be low since
real-world data is often intrinsically non-linear. To combine both the benefits
of the high predictive performance of DNNs and high interpretability of linear
models into a single model, we propose neural generators of sparse local linear
models (NGSLLs). The sparse local linear models have high flexibility as they
can approximate non-linear functions. The NGSLL generates sparse linear weights
for each sample using DNNs that take original representations of each sample
(e.g., word sequence) and their simplified representations (e.g., bag-of-words)
as input. By extracting features from the original representations, the weights
can contain rich information to achieve high predictive performance.
Additionally, the prediction is interpretable because it is obtained by the
inner product between the simplified representations and the sparse weights,
where only a small number of weights are selected by our gate module in the
NGSLL. In experiments with real-world datasets, we demonstrate the
effectiveness of the NGSLL quantitatively and qualitatively by evaluating
prediction performance and visualizing generated weights on image and text
classification tasks.

positive:
The practical impact of deep learning on complex supervised learning problems
has been significant, so much so that almost every Artificial Intelligence
problem, or at least a portion thereof, has been somehow recast as a deep
learning problem. The applications appeal is significant, but this appeal is
increasingly challenged by what some call the challenge of explainability, or
more generally the more traditional challenge of debuggability: if the outcomes
of a deep learning process produce unexpected results (e.g., less than expected
performance of a classifier), then there is little available in the way of
theories or tools to help investigate the potential causes of such unexpected
behavior, especially when this behavior could impact people's lives. We
describe a preliminary framework to help address this issue, which we call
"deep visual explanation" (DVE). "Deep," because it is the development and
performance of deep neural network models that we want to understand. "Visual,"
because we believe that the most rapid insight into a complex multi-dimensional
model is provided by appropriate visualization techniques, and "Explanation,"
because in the spectrum from instrumentation by inserting print statements to
the abductive inference of explanatory hypotheses, we believe that the key to
understanding deep learning relies on the identification and exposure of
hypotheses about the performance behavior of a learned deep model. In the
exposition of our preliminary framework, we use relatively straightforward
image classification examples and a variety of choices on initial configuration
of a deep model building scenario. By careful but not complicated
instrumentation, we expose classification outcomes of deep models using
visualization, and also show initial results for one potential application of
interpretability.

positive:
Automated machine learning (AutoML) usually involves several crucial
components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization
(HPO), and Neural Architecture Search (NAS). Although many strategies have been
developed for automating these components in separation, joint optimization of
these components remains challenging due to the largely increased search
dimension and the variant input types of each component. In parallel to this,
the common practice of searching for the optimal architecture first and then
retraining it before deployment in NAS often suffers from low performance
correlation between the searching and retraining stages. An end-to-end solution
that integrates the AutoML components and returns a ready-to-use model at the
end of the search is desirable. In view of these, we propose DHA, which
achieves joint optimization of Data augmentation policy, Hyper-parameter and
Architecture. Specifically, end-to-end NAS is achieved in a differentiable
manner by optimizing a compressed lower-dimensional feature space, while DA
policy and HPO are regarded as dynamic schedulers, which adapt themselves to
the update of network parameters and network architecture at the same time.
Experiments show that DHA achieves state-of-the-art (SOTA) results on various
datasets and search spaces. To the best of our knowledge, we are the first to
efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end
manner without retraining.

positive:
With the availability of large databases and recent improvements in deep
learning methodology, the performance of AI systems is reaching or even
exceeding the human level on an increasing number of complex tasks. Impressive
examples of this development can be found in domains such as image
classification, sentiment analysis, speech understanding or strategic game
playing. However, because of their nested non-linear structure, these highly
successful machine learning and artificial intelligence models are usually
applied in a black box manner, i.e., no information is provided about what
exactly makes them arrive at their predictions. Since this lack of transparency
can be a major drawback, e.g., in medical applications, the development of
methods for visualizing, explaining and interpreting deep learning models has
recently attracted increasing attention. This paper summarizes recent
developments in this field and makes a plea for more interpretability in
artificial intelligence. Furthermore, it presents two approaches to explaining
predictions of deep learning models, one method which computes the sensitivity
of the prediction with respect to changes in the input and one approach which
meaningfully decomposes the decision in terms of the input variables. These
methods are evaluated on three classification tasks.

positive:
The ability of deep learning (DL) to improve the practice of medicine and its
clinical outcomes faces a looming obstacle: model interpretation. Without
description of how outputs are generated, a collaborating physician can neither
resolve when the model's conclusions are in conflict with his or her own, nor
learn to anticipate model behavior. Current research aims to interpret networks
that diagnose ECG recordings, which has great potential impact as recordings
become more personalized and widely deployed. A generalizable impact beyond
ECGs lies in the ability to provide a rich test-bed for the development of
interpretive techniques in medicine. Interpretive techniques for Deep Neural
Networks (DNNs), however, tend to be heuristic and observational in nature,
lacking the mathematical rigor one might expect in the analysis of math
equations. The motivation of this paper is to offer a third option, a
scientific approach. We treat the model output itself as a phenomenon to be
explained through component parts and equations governing their behavior. We
argue that these component parts should also be "black boxes" --additional
targets to interpret heuristically with clear functional connection to the
original. We show how to rigorously factor a DNN into a hierarchical equation
consisting of black box variables. This is not a subdivision into physical
parts, like an organism into its cells; it is but one choice of an equation
into a collection of abstract functions. Yet, for DNNs trained to identify
normal ECG waveforms on PhysioNet 2017 Challenge data, we demonstrate this
choice yields interpretable component models identified with visual composite
sketches of ECG samples in corresponding input regions. Moreover, the recursion
distills this interpretation: additional factorization of component black boxes
corresponds to ECG partitions that are more morphologically pure.

positive:
Model interpretability is a requirement in many applications in which crucial
decisions are made by users relying on a model's outputs. The recent movement
for "algorithmic fairness" also stipulates explainability, and therefore
interpretability of learning models. And yet the most successful contemporary
Machine Learning approaches, the Deep Neural Networks, produce models that are
highly non-interpretable. We attempt to address this challenge by proposing a
technique called CNN-INTE to interpret deep Convolutional Neural Networks (CNN)
via meta-learning. In this work, we interpret a specific hidden layer of the
deep CNN model on the MNIST image dataset. We use a clustering algorithm in a
two-level structure to find the meta-level training data and Random Forest as
base learning algorithms to generate the meta-level test data. The
interpretation results are displayed visually via diagrams, which clearly
indicates how a specific test instance is classified. Our method achieves
global interpretation for all the test instances without sacrificing the
accuracy obtained by the original deep CNN model. This means our model is
faithful to the deep CNN model, which leads to reliable interpretations.

positive:
With the unprecedented proliferation of machine learning software, there is
an ever-increasing need to generate efficient code for such applications.
State-of-the-art deep-learning compilers like TVM and Halide incorporate a
learning-based performance model to search the space of valid implementations
of a given deep learning algorithm. For a given application, the model
generates a performance metric such as the run time without executing the
application on hardware. Such models speed up the compilation process by
obviating the need to benchmark an enormous number of candidate
implementations, referred to as schedules, on hardware. Existing performance
models employ feed-forward networks, recurrent networks, or decision tree
ensembles to estimate the performance of different implementations of a neural
network. Graphs present a natural and intuitive way to model deep-learning
networks where each node represents a computational stage or operation.
Incorporating the inherent graph structure of these workloads in the
performance model can enable a better representation and learning of
inter-stage interactions. The accuracy of a performance model has direct
implications on the efficiency of the search strategy, making it a crucial
component of this class of deep-learning compilers. In this work, we develop a
novel performance model that adopts a graph representation. In our model, each
stage of computation represents a node characterized by features that capture
the operations performed by the stage. The interaction between nodes is
achieved using graph convolutions. Experimental evaluation shows a 7:75x and
12x reduction in prediction error compared to the Halide and TVM models,
respectively.

positive:
Interpretable machine learning has become a strong competitor for traditional
black-box models. However, the possible loss of the predictive performance for
gaining interpretability is often inevitable, putting practitioners in a
dilemma of choosing between high accuracy (black-box models) and
interpretability (interpretable models). In this work, we propose a novel
framework for building a Hybrid Predictive Model (HPM) that integrates an
interpretable model with any black-box model to combine their strengths. The
interpretable model substitutes the black-box model on a subset of data where
the black-box is overkill or nearly overkill, gaining transparency at no or low
cost of the predictive accuracy. We design a principled objective function that
considers predictive accuracy, model interpretability, and model transparency
(defined as the percentage of data processed by the interpretable substitute.)
Under this framework, we propose two hybrid models, one substituting with
association rules and the other with linear models, and we design customized
training algorithms for both models. We test the hybrid models on structured
data and text data where interpretable models collaborate with various
state-of-the-art black-box models. Results show that hybrid models obtain an
efficient trade-off between transparency and predictive performance,
characterized by our proposed efficient frontiers.

positive:
State of the art machine learning algorithms are highly optimized to provide
the optimal prediction possible, naturally resulting in complex models. While
these models often outperform simpler more interpretable models by order of
magnitudes, in terms of understanding the way the model functions, we are often
facing a "black box".
  In this paper we suggest a simple method to interpret the behavior of any
predictive model, both for regression and classification. Given a particular
model, the information required to interpret it can be obtained by studying the
partial derivatives of the model with respect to the input. We exemplify this
insight by interpreting convolutional and multi-layer neural networks in the
field of natural language processing.

positive:
Recent advances in deep learning have achieved impressive gains in
classification accuracy on a variety of types of data, including images and
text. Despite these gains, however, concerns have been raised about the
calibration, robustness, and interpretability of these models. In this paper we
propose a simple way to modify any conventional deep architecture to
automatically provide more transparent explanations for classification
decisions, as well as an intuitive notion of the credibility of each
prediction. Specifically, we draw on ideas from nonparametric kernel
regression, and propose to predict labels based on a weighted sum of training
instances, where the weights are determined by distance in a learned
instance-embedding space. Working within the framework of conformal methods, we
propose a new measure of nonconformity suggested by our model, and
experimentally validate the accompanying theoretical expectations,
demonstrating improved transparency, controlled error rates, and robustness to
out-of-domain data, without compromising on accuracy or calibration.

positive:
The work concerns the problem of reducing a pre-trained deep neuronal network
to a smaller network, with just few layers, whilst retaining the network's
functionality on a given task
  The proposed approach is motivated by the observation that the aim to deliver
the highest accuracy possible in the broadest range of operational conditions,
which many deep neural networks models strive to achieve, may not necessarily
be always needed, desired, or even achievable due to the lack of data or
technical constraints. In relation to the face recognition problem, we
formulated an example of such a usecase, the `backyard dog' problem. The
`backyard dog', implemented by a lean network, should correctly identify
members from a limited group of individuals, a `family', and should distinguish
between them. At the same time, the network must produce an alarm to an image
of an individual who is not in a member of the family. To produce such a
network, we propose a shallowing algorithm. The algorithm takes an existing
deep learning model on its input and outputs a shallowed version of it. The
algorithm is non-iterative and is based on the Advanced Supervised Principal
Component Analysis. Performance of the algorithm is assessed in exhaustive
numerical experiments. In the above usecase, the `backyard dog' problem, the
method is capable of drastically reducing the depth of deep learning neural
networks, albeit at the cost of mild performance deterioration.
  We developed a simple non-iterative method for shallowing down pre-trained
deep networks. The method is generic in the sense that it applies to a broad
class of feed-forward networks, and is based on the Advanced Supervise
Principal Component Analysis. The method enables generation of families of
smaller-size shallower specialized networks tuned for specific operational
conditions and tasks from a single larger and more universal legacy network.

positive:
Due to the capability of deep learning to perform well in high dimensional
problems, deep reinforcement learning agents perform well in challenging tasks
such as Atari 2600 games. However, clearly explaining why a certain action is
taken by the agent can be as important as the decision itself. Deep
reinforcement learning models, as other deep learning models, tend to be opaque
in their decision-making process. In this work, we propose to make deep
reinforcement learning more transparent by visualizing the evidence on which
the agent bases its decision. In this work, we emphasize the importance of
producing a justification for an observed action, which could be applied to a
black-box decision agent.

positive:
Deep learning has become popular because of its potential to achieve high
accuracy in prediction tasks. However, accuracy is not always the only goal of
statistical modelling, especially for models developed as part of scientific
research. Rather, many scientific models are developed to facilitate scientific
discovery, by which we mean to abstract a human-understandable representation
of the natural world. Unfortunately, the opacity of deep neural networks limit
their role in scientific discovery, creating a new demand for models that are
transparently interpretable. This article is a field guide to transparent model
design. It provides a taxonomy of transparent model design concepts, a
practical workflow for putting design concepts into practice, and a general
template for reporting design choices. We hope this field guide will help
researchers more effectively design transparently interpretable models, and
thus enable them to use deep learning for scientific discovery.

positive:
In recent years, machine learning (ML) has become a key enabling technology
for the sciences and industry. Especially through improvements in methodology,
the availability of large databases and increased computational power, today's
ML algorithms are able to achieve excellent performance (at times even
exceeding the human level) on an increasing number of complex tasks. Deep
learning models are at the forefront of this development. However, due to their
nested non-linear structure, these powerful models have been generally
considered "black boxes", not providing any information about what exactly
makes them arrive at their predictions. Since in many applications, e.g., in
the medical domain, such lack of transparency may be not acceptable, the
development of methods for visualizing, explaining and interpreting deep
learning models has recently attracted increasing attention. This introductory
paper presents recent developments and applications in this field and makes a
plea for a wider use of explainable learning algorithms in practice.

positive:
Deep neural networks are learning models with a very high capacity and
therefore prone to over-fitting. Many regularization techniques such as
Dropout, DropConnect, and weight decay all attempt to solve the problem of
over-fitting by reducing the capacity of their respective models (Srivastava et
al., 2014), (Wan et al., 2013), (Krogh & Hertz, 1992). In this paper we
introduce a new form of regularization that guides the learning problem in a
way that reduces over-fitting without sacrificing the capacity of the model.
The mistakes that models make in early stages of training carry information
about the learning problem. By adjusting the labels of the current epoch of
training through a weighted average of the real labels, and an exponential
average of the past soft-targets we achieved a regularization scheme as
powerful as Dropout without necessarily reducing the capacity of the model, and
simplified the complexity of the learning problem. SoftTarget regularization
proved to be an effective tool in various neural network architectures.

positive:
We propose a possible solution to a public challenge posed by the Fair Isaac
Corporation (FICO), which is to provide an explainable model for credit risk
assessment. Rather than present a black box model and explain it afterwards, we
provide a globally interpretable model that is as accurate as other neural
networks. Our "two-layer additive risk model" is decomposable into subscales,
where each node in the second layer represents a meaningful subscale, and all
of the nonlinearities are transparent. We provide three types of explanations
that are simpler than, but consistent with, the global model. One of these
explanation methods involves solving a minimum set cover problem to find
high-support globally-consistent explanations. We present a new online
visualization tool to allow users to explore the global model and its
explanations.

positive:
Physical motion models offer interpretable predictions for the motion of
vehicles. However, some model parameters, such as those related to aero- and
hydrodynamics, are expensive to measure and are often only roughly approximated
reducing prediction accuracy. Recurrent neural networks achieve high prediction
accuracy at low cost, as they can use cheap measurements collected during
routine operation of the vehicle, but their results are hard to interpret. To
precisely predict vehicle states without expensive measurements of physical
parameters, we propose a hybrid approach combining deep learning and physical
motion models including a novel two-phase training procedure. We achieve
interpretability by restricting the output range of the deep neural network as
part of the hybrid model, which limits the uncertainty introduced by the neural
network to a known quantity. We have evaluated our approach for the use case of
ship and quadcopter motion. The results show that our hybrid model can improve
model interpretability with no decrease in accuracy compared to existing deep
learning approaches.

positive:
While deep learning has achieved great success in many fields, one common
criticism about deep learning is its lack of interpretability. In most cases,
the hidden units in a deep neural network do not have a clear semantic meaning
or correspond to any physical entities. However, model interpretability and
explainability are crucial in many biomedical applications. To address this
challenge, we developed the Factor Graph Neural Network model that is
interpretable and predictable by combining probabilistic graphical models with
deep learning. We directly encode biological knowledge such as Gene Ontology as
a factor graph into the model architecture, making the model transparent and
interpretable. Furthermore, we devised an attention mechanism that can capture
multi-scale hierarchical interactions among biological entities such as genes
and Gene Ontology terms. With parameter sharing mechanism, the unrolled Factor
Graph Neural Network model can be trained with stochastic depth and generalize
well. We applied our model to two cancer genomic datasets to predict target
clinical variables and achieved better results than other traditional machine
learning and deep learning models. Our model can also be used for gene set
enrichment analysis and selecting Gene Ontology terms that are important to
target clinical variables.

positive:
In many machine learning tasks, learning a good representation of the data
can be the key to building a well-performant solution. This is because most
learning algorithms operate with the features in order to find models for the
data. For instance, classification performance can improve if the data is
mapped to a space where classes are easily separated, and regression can be
facilitated by finding a manifold of data in the feature space. As a general
rule, features are transformed by means of statistical methods such as
principal component analysis, or manifold learning techniques such as Isomap or
locally linear embedding. From a plethora of representation learning methods,
one of the most versatile tools is the autoencoder. In this paper we aim to
demonstrate how to influence its learned representations to achieve the desired
learning behavior. To this end, we present a series of learning tasks: data
embedding for visualization, image denoising, semantic hashing, detection of
abnormal behaviors and instance generation. We model them from the
representation learning perspective, following the state of the art
methodologies in each field. A solution is proposed for each task employing
autoencoders as the only learning method. The theoretical developments are put
into practice using a selection of datasets for the different problems and
implementing each solution, followed by a discussion of the results in each
case study and a brief explanation of other six learning applications. We also
explore the current challenges and approaches to explainability in the context
of autoencoders. All of this helps conclude that, thanks to alterations in
their structure as well as their objective function, autoencoders may be the
core of a possible solution to many problems which can be modeled as a
transformation of the feature space.

positive:
Machine learning has recently been widely adopted to address the managerial
decision making problems, in which the decision maker needs to be able to
interpret the contributions of individual attributes in an explicit form.
However, there is a trade-off between performance and interpretability. Full
complexity models are non-traceable black-box, whereas classic interpretable
models are usually simplified with lower accuracy. This trade-off limits the
application of state-of-the-art machine learning models in management problems,
which requires high prediction performance, as well as the understanding of
individual attributes' contributions to the model outcome. Multiple criteria
decision aiding (MCDA) is a family of analytic approaches to depicting the
rationale of human decision. It is also limited by strong assumptions. To meet
the decision maker's demand for more interpretable machine learning models, we
propose a novel hybrid method, namely Neural Network-based Multiple Criteria
Decision Aiding, which combines an additive value model and a fully-connected
multilayer perceptron (MLP) to achieve good performance while capturing the
explicit relationships between individual attributes and the prediction.
NN-MCDA has a linear component to characterize such relationships through
providing explicit marginal value functions, and a nonlinear component to
capture the implicit high-order interactions between attributes and their
complex nonlinear transformations. We demonstrate the effectiveness of NN-MCDA
with extensive simulation studies and three real-world datasets. To the best of
our knowledge, this research is the first to enhance the interpretability of
machine learning models with MCDA techniques. The proposed framework also sheds
light on how to use machine learning techniques to free MCDA from strong
assumptions.

positive:
Contemporary deep learning based video captioning follows encoder-decoder
framework. In encoder, visual features are extracted with 2D/3D Convolutional
Neural Networks (CNNs) and a transformed version of those features is passed to
the decoder. The decoder uses word embeddings and a language model to map
visual features to natural language captions. Due to its composite nature, the
encoder-decoder pipeline provides the freedom of multiple choices for each of
its components, e.g the choices of CNNs models, feature transformations, word
embeddings, and language models etc. Component selection can have drastic
effects on the overall video captioning performance. However, current
literature is void of any systematic investigation in this regard. This article
fills this gap by providing the first thorough empirical analysis of the role
that each major component plays in a contemporary video captioning pipeline. We
perform extensive experiments by varying the constituent components of the
video captioning framework, and quantify the performance gains that are
possible by mere component selection. We use the popular MSVD dataset as the
test-bed, and demonstrate that substantial performance gains are possible by
careful selection of the constituent components without major changes to the
pipeline itself. These results are expected to provide guiding principles for
future research in the fast growing direction of video captioning.

positive:
Among interpretable machine learning methods, the class of Generalised
Additive Neural Networks (GANNs) is referred to as Self-Explaining Neural
Networks (SENN) because of the linear dependence on explicit functions of the
inputs. In binary classification this shows the precise weight that each input
contributes towards the logit. The nomogram is a graphical representation of
these weights. We show that functions of individual and pairs of variables can
be derived from a functional Analysis of Variance (ANOVA) representation,
enabling an efficient feature selection to be carried by application of the
logistic Lasso. This process infers the structure of GANNs which otherwise
needs to be predefined. As this method is particularly suited for tabular data,
it starts by fitting a generic flexible model, in this case a Multi-layer
Perceptron (MLP) to which the ANOVA decomposition is applied. This has the
further advantage that the resulting GANN can be replicated as a SENN, enabling
further refinement of the univariate and bivariate component functions to take
place. The component functions are partial responses hence the SENN is a
partial response network. The Partial Response Network (PRN) is equally as
transparent as a traditional logistic regression model, but capable of
non-linear classification with comparable or superior performance to the
original MLP. In other words, the PRN is a fully interpretable representation
of the MLP, at the level of univariate and bivariate effects. The performance
of the PRN is shown to be competitive for benchmark data, against
state-of-the-art machine learning methods including GBM, SVM and Random
Forests. It is also compared with spline-based Sparse Additive Models (SAM)
showing that a semi-parametric representation of the GAM as a neural network
can be as effective as the SAM though less constrained by the need to set
spline nodes.

positive:
Deep Neural Networks have achieved huge success at a wide spectrum of
applications from language modeling, computer vision to speech recognition.
However, nowadays, good performance alone is not sufficient to satisfy the
needs of practical deployment where interpretability is demanded for cases
involving ethics and mission critical applications. The complex models of Deep
Neural Networks make it hard to understand and reason the predictions, which
hinders its further progress. To tackle this problem, we apply the Knowledge
Distillation technique to distill Deep Neural Networks into decision trees in
order to attain good performance and interpretability simultaneously. We
formulate the problem at hand as a multi-output regression problem and the
experiments demonstrate that the student model achieves significantly better
accuracy performance (about 1\% to 5\%) than vanilla decision trees at the same
level of tree depth. The experiments are implemented on the TensorFlow platform
to make it scalable to big datasets. To the best of our knowledge, we are the
first to distill Deep Neural Networks into vanilla decision trees on
multi-class datasets.

positive:
The black-box nature of deep learning models prevents them from being
completely trusted in domains like biomedicine. Most explainability techniques
do not capture the concept-based reasoning that human beings follow. In this
work, we attempt to understand the behavior of trained models that perform
image processing tasks in the medical domain by building a graphical
representation of the concepts they learn. Extracting such a graphical
representation of the model's behavior on an abstract, higher conceptual level
would unravel the learnings of these models and would help us to evaluate the
steps taken by the model for predictions. We show the application of our
proposed implementation on two biomedical problems - brain tumor segmentation
and fundus image classification. We provide an alternative graphical
representation of the model by formulating a concept level graph as discussed
above, which makes the problem of intervention to find active inference trails
more tractable. Understanding these trails would provide an understanding of
the hierarchy of the decision-making process followed by the model. [As well as
overall nature of model]. Our framework is available at
https://github.com/koriavinash1/BioExp

positive:
Despite outperforming the human in many tasks, deep neural network models are
also criticized for the lack of transparency and interpretability in decision
making. The opaqueness results in uncertainty and low confidence when deploying
such a model in model sharing scenarios, when the model is developed by a third
party. For a supervised machine learning model, sharing training process
including training data provides an effective way to gain trust and to better
understand model predictions. However, it is not always possible to share all
training data due to privacy and policy constraints. In this paper, we propose
a method to disclose a small set of training data that is just sufficient for
users to get the insight of a complicated model. The method constructs a
boundary tree using selected training data and the tree is able to approximate
the complicated model with high fidelity. We show that traversing data points
in the tree gives users significantly better understanding of the model and
paves the way for trustworthy model sharing.

negative:
The widespread use of deep neural networks has achieved substantial success
in many tasks. However, there still exists a huge gap between the operating
mechanism of deep learning models and human-understandable decision making, so
that humans cannot fully trust the predictions made by these models. To date,
little work has been done on how to align the behaviors of deep learning models
with human perception in order to train a human-understandable model. To fill
this gap, we propose a new framework to train a deep neural network by
incorporating the prior of human perception into the model learning process.
Our proposed model mimics the process of perceiving conceptual parts from
images and assessing their relative contributions towards the final
recognition. The effectiveness of our proposed model is evaluated on two
classical visual recognition tasks. The experimental results and analysis
confirm our model is able to provide interpretable explanations for its
predictions, but also maintain competitive recognition accuracy.

negative:
Is a deep learning model capable of understanding systems governed by certain
first principle laws by only observing the system's output? Can deep learning
learn the underlying physics and honor the physics when making predictions? The
answers are both positive. In an effort to simulate two-dimensional subsurface
fluid dynamics in porous media, we found that an accurate deep-learning-based
proxy model can be taught efficiently by a computationally expensive
finite-volume-based simulator. We pose the problem as an image-to-image
regression, running the simulator with different input parameters to furnish a
synthetic training dataset upon which we fit the deep learning models. Since
the data is spatiotemporal, we compare the performance of two alternative
treatments of time; a convolutional LSTM versus an autoencoder network that
treats time as a direct input. Adversarial methods are adopted to address the
sharp spatial gradient in the fluid dynamic problems. Compared to traditional
simulation, the proposed deep learning approach enables much faster forward
computation, which allows us to explore more scenarios with a much larger
parameter space given the same time. It is shown that the improved forward
computation efficiency is particularly valuable in solving inversion problems,
where the physics model has unknown parameters to be determined by history
matching. By computing the pixel-level attention of the trained model, we
quantify the sensitivity of the deep learning model to key physical parameters
and hence demonstrate that the inversion problems can be solved with great
acceleration. We assess the efficacy of the machine learning surrogate in terms
of its training speed and accuracy. The network can be trained within minutes
using limited training data and achieve accuracy that scales desirably with the
amount of training data supplied.

negative:
Virtual reality (VR) offers immersive visualization and intuitive
interaction. We leverage VR to enable any biomedical professional to deploy a
deep learning (DL) model for image classification. While DL models can be
powerful tools for data analysis, they are also challenging to understand and
develop. To make deep learning more accessible and intuitive, we have built a
virtual reality-based DL development environment. Within our environment, the
user can move tangible objects to construct a neural network only using their
hands. Our software automatically translates these configurations into a
trainable model and then reports its resulting accuracy on a test dataset in
real-time. Furthermore, we have enriched the virtual objects with
visualizations of the model's components such that users can achieve insight
about the DL models that they are developing. With this approach, we bridge the
gap between professionals in different fields of expertise while offering a
novel perspective for model analysis and data interaction. We further suggest
that techniques of development and visualization in deep learning can benefit
by integrating virtual reality.

negative:
Histopathology image analysis can be considered as a Multiple instance
learning (MIL) problem, where the whole slide histopathology image (WSI) is
regarded as a bag of instances (i.e, patches) and the task is to predict a
single class label to the WSI. However, in many real-life applications such as
computational pathology, discovering the key instances that trigger the bag
label is of great interest because it provides reasons for the decision made by
the system. In this paper, we propose a deep convolutional neural network (CNN)
model that addresses the primary task of a bag classification on a WSI and also
learns to identify the response of each instance to provide interpretable
results to the final prediction. We incorporate the attention mechanism into
the proposed model to operate the transformation of instances and learn
attention weights to allow us to find key patches. To perform a balanced
training, we introduce adaptive weighing in each training bag to explicitly
adjust the weight distribution in order to concentrate more on the contribution
of hard samples. Based on the learned attention weights, we further develop a
solution to boost the classification performance by generating the bags with
hard negative instances. We conduct extensive experiments on colon and breast
cancer histopathology data and show that our framework achieves
state-of-the-art performance.

negative:
Deep learning is very effective at jointly learning feature representations
and classification models, especially when dealing with high dimensional input
patterns. Probabilistic logic reasoning, on the other hand, is capable to take
consistent and robust decisions in complex environments. The integration of
deep learning and logic reasoning is still an open-research problem and it is
considered to be the key for the development of real intelligent agents. This
paper presents Deep Logic Models, which are deep graphical models integrating
deep learning and logic reasoning both for learning and inference. Deep Logic
Models create an end-to-end differentiable architecture, where deep learners
are embedded into a network implementing a continuous relaxation of the logic
knowledge. The learning process allows to jointly learn the weights of the deep
learners and the meta-parameters controlling the high-level reasoning. The
experimental results show that the proposed methodology overtakes the
limitations of the other approaches that have been proposed to bridge deep
learning and reasoning.

negative:
Many seemingly unrelated computer vision tasks can be viewed as a special
case of image decomposition into separate layers. For example, image
segmentation (separation into foreground and background layers); transparent
layer separation (into reflection and transmission layers); Image dehazing
(separation into a clear image and a haze map), and more. In this paper we
propose a unified framework for unsupervised layer decomposition of a single
image, based on coupled "Deep-image-Prior" (DIP) networks. It was shown
[Ulyanov et al] that the structure of a single DIP generator network is
sufficient to capture the low-level statistics of a single image. We show that
coupling multiple such DIPs provides a powerful tool for decomposing images
into their basic components, for a wide variety of applications. This
capability stems from the fact that the internal statistics of a mixture of
layers is more complex than the statistics of each of its individual
components. We show the power of this approach for Image-Dehazing, Fg/Bg
Segmentation, Watermark-Removal, Transparency Separation in images and video,
and more. These capabilities are achieved in a totally unsupervised way, with
no training examples other than the input image/video itself.

negative:
We present an interpretable companion model for any pre-trained black-box
classifiers. The idea is that for any input, a user can decide to either
receive a prediction from the black-box model, with high accuracy but no
explanations, or employ a companion rule to obtain an interpretable prediction
with slightly lower accuracy. The companion model is trained from data and the
predictions of the black-box model, with the objective combining area under the
transparency--accuracy curve and model complexity. Our model provides flexible
choices for practitioners who face the dilemma of choosing between always using
interpretable models and always using black-box models for a predictive task,
so users can, for any given input, take a step back to resort to an
interpretable prediction if they find the predictive performance satisfying, or
stick to the black-box model if the rules are unsatisfying. To show the value
of companion models, we design a human evaluation on more than a hundred people
to investigate the tolerable accuracy loss to gain interpretability for humans.

negative:
We propose Black Box Explanations through Transparent Approximations (BETA),
a novel model agnostic framework for explaining the behavior of any black-box
classifier by simultaneously optimizing for fidelity to the original model and
interpretability of the explanation. To this end, we develop a novel objective
function which allows us to learn (with optimality guarantees), a small number
of compact decision sets each of which explains the behavior of the black box
model in unambiguous, well-defined regions of feature space. Furthermore, our
framework also is capable of accepting user input when generating these
approximations, thus allowing users to interactively explore how the black-box
model behaves in different subspaces that are of interest to the user. To the
best of our knowledge, this is the first approach which can produce global
explanations of the behavior of any given black box model through joint
optimization of unambiguity, fidelity, and interpretability, while also
allowing users to explore model behavior based on their preferences.
Experimental evaluation with real-world datasets and user studies demonstrates
that our approach can generate highly compact, easy-to-understand, yet accurate
approximations of various kinds of predictive models compared to
state-of-the-art baselines.

negative:
Interpretable machine learning (IML) becomes increasingly important in highly
regulated industry sectors related to the health and safety or fundamental
rights of human beings. In general, the inherently IML models should be adopted
because of their transparency and explainability, while black-box models with
model-agnostic explainability can be more difficult to defend under regulatory
scrutiny. For assessing inherent interpretability of a machine learning model,
we propose a qualitative template based on feature effects and model
architecture constraints. It provides the design principles for
high-performance IML model development, with examples given by reviewing our
recent works on ExNN, GAMI-Net, SIMTree, and the Aletheia toolkit for local
linear interpretability of deep ReLU networks. We further demonstrate how to
design an interpretable ReLU DNN model with evaluation of conceptual soundness
for a real case study of predicting credit default in home lending. We hope
that this work will provide a practical guide of developing inherently IML
models in high risk applications in banking industry, as well as other sectors.

negative:
As the discipline has evolved, research in machine learning has been focused
more and more on creating more powerful neural networks, without regard for the
interpretability of these networks. Such "black-box models" yield
state-of-the-art results, but we cannot understand why they make a particular
decision or prediction. Sometimes this is acceptable, but often it is not. We
propose a novel architecture, Regression Networks, which combines the power of
neural networks with the understandability of regression analysis. While some
methods for combining these exist in the literature, our architecture
generalizes these approaches by taking interactions into account, offering the
power of a dense neural network without forsaking interpretability. We
demonstrate that the models exceed the state-of-the-art performance of
interpretable models on several benchmark datasets, matching the power of a
dense neural network. Finally, we discuss how these techniques can be
generalized to other neural architectures, such as convolutional and recurrent
neural networks.

negative:
Nonlinear methods such as Deep Neural Networks (DNNs) are the gold standard
for various challenging machine learning problems, e.g., image classification,
natural language processing or human action recognition. Although these methods
perform impressively well, they have a significant disadvantage, the lack of
transparency, limiting the interpretability of the solution and thus the scope
of application in practice. Especially DNNs act as black boxes due to their
multilayer nonlinear structure. In this paper we introduce a novel methodology
for interpreting generic multilayer neural networks by decomposing the network
classification decision into contributions of its input elements. Although our
focus is on image classification, the method is applicable to a broad set of
input data, learning tasks and network architectures. Our method is based on
deep Taylor decomposition and efficiently utilizes the structure of the network
by backpropagating the explanations from the output to the input layer. We
evaluate the proposed method empirically on the MNIST and ILSVRC data sets.

negative:
Deep neural networks have been well-known for their superb handling of
various machine learning and artificial intelligence tasks. However, due to
their over-parameterized black-box nature, it is often difficult to understand
the prediction results of deep models. In recent years, many interpretation
tools have been proposed to explain or reveal how deep models make decisions.
In this paper, we review this line of research and try to make a comprehensive
survey. Specifically, we first introduce and clarify two basic concepts --
interpretations and interpretability -- that people usually get confused about.
To address the research efforts in interpretations, we elaborate the designs of
a number of interpretation algorithms, from different perspectives, by
proposing a new taxonomy. Then, to understand the interpretation results, we
also survey the performance metrics for evaluating interpretation algorithms.
Further, we summarize the current works in evaluating models' interpretability
using "trustworthy" interpretation algorithms. Finally, we review and discuss
the connections between deep models' interpretations and other factors, such as
adversarial robustness and learning from interpretations, and we introduce
several open-source libraries for interpretation algorithms and evaluation
approaches.

negative:
Interpretability and explainability of deep neural networks are challenging
due to their scale, complexity, and the agreeable notions on which the
explaining process rests. Previous work, in particular, has focused on
representing internal components of neural networks through human-friendly
visuals and concepts. On the other hand, in real life, when making a decision,
human tends to rely on similar situations and/or associations in the past.
Hence arguably, a promising approach to make the model transparent is to design
it in a way such that the model explicitly connects the current sample with the
seen ones, and bases its decision on these samples. Grounded on that principle,
we propose in this paper an explainable, evidence-based memory network
architecture, which learns to summarize the dataset and extract supporting
evidences to make its decision. Our model achieves state-of-the-art performance
on two popular question answering datasets (i.e. TrecQA and WikiQA). Via
further analysis, we show that this model can reliably trace the errors it has
made in the validation step to the training instances that might have caused
these errors. We believe that this error-tracing capability provides
significant benefit in improving dataset quality in many applications.

negative:
Deep Learning algorithms are often used as black box type learning and they
are too complex to understand. The widespread usability of Deep Learning
algorithms to solve various machine learning problems demands deep and
transparent understanding of the internal representation as well as decision
making. Moreover, the learning models, trained on sequential data, such as
audio and video data, have intricate internal reasoning process due to their
complex distribution of features. Thus, a visual simulator might be helpful to
trace the internal decision making mechanisms in response to adversarial input
data, and it would help to debug and design appropriate deep learning models.
However, interpreting the internal reasoning of deep learning model is not well
studied in the literature. In this work, we have developed a visual interactive
web application, namely d-DeVIS, which helps to visualize the internal
reasoning of the learning model which is trained on the audio data. The
proposed system allows to perceive the behavior as well as to debug the model
by interactively generating adversarial audio data point. The web application
of d-DeVIS is available at ddevis.herokuapp.com.

negative:
Credit assessments activities are essential for financial institutions and
allow the global economy to grow. Building robust, solid and accurate models
that estimate the probability of a default of a company is mandatory for credit
insurance companies, moreover when it comes to bridging the trade finance gap.
Automating the risk assessment process will allow credit risk experts to reduce
their workload and focus on the critical and complex cases, as well as to
improve the loan approval process by reducing the time to process the
application. The recent developments in Artificial Intelligence are offering
new powerful opportunities. However, most AI techniques are labelled as
blackbox models due to their lack of explainability. For both users and
regulators, in order to deploy such technologies at scale, being able to
understand the model logic is a must to grant accurate and ethical decision
making. In this study, we focus on companies credit scoring and we benchmark
different machine learning models. The aim is to build a model to predict
whether a company will experience financial problems in a given time horizon.
We address the black box problem using eXplainable Artificial Techniques in
particular, post-hoc explanations using SHapley Additive exPlanations. We bring
light by providing an expert-aligned feature relevance score highlighting the
disagreement between a credit risk expert and a model feature attribution
explanation in order to better quantify the convergence towards a better
human-aligned decision making.

negative:
Machine learning solutions for pattern classification problems are nowadays
widely deployed in society and industry. However, the lack of transparency and
accountability of most accurate models often hinders their safe use. Thus,
there is a clear need for developing explainable artificial intelligence
mechanisms. There exist model-agnostic methods that summarize feature
contributions, but their interpretability is limited to predictions made by
black-box models. An open challenge is to develop models that have intrinsic
interpretability and produce their own explanations, even for classes of models
that are traditionally considered black boxes like (recurrent) neural networks.
In this paper, we propose a Long-Term Cognitive Network for interpretable
pattern classification of structured data. Our method brings its own mechanism
for providing explanations by quantifying the relevance of each feature in the
decision process. For supporting the interpretability without affecting the
performance, the model incorporates more flexibility through a quasi-nonlinear
reasoning rule that allows controlling nonlinearity. Besides, we propose a
recurrence-aware decision model that evades the issues posed by the unique
fixed point while introducing a deterministic learning algorithm to compute the
tunable parameters. The simulations show that our interpretable model obtains
competitive results when compared to state-of-the-art white and black-box
models.

negative:
Deep learning has been widely used within learning algorithms for robotics.
One disadvantage of deep networks is that these networks are black-box
representations. Therefore, the learned approximations ignore the existing
knowledge of physics or robotics. Especially for learning dynamics models,
these black-box models are not desirable as the underlying principles are well
understood and the standard deep networks can learn dynamics that violate these
principles. To learn dynamics models with deep networks that guarantee
physically plausible dynamics, we introduce physics-inspired deep networks that
combine first principles from physics with deep learning. We incorporate
Lagrangian mechanics within the model learning such that all approximated
models adhere to the laws of physics and conserve energy. Deep Lagrangian
Networks (DeLaN) parametrize the system energy using two networks. The
parameters are obtained by minimizing the squared residual of the
Euler-Lagrange differential equation. Therefore, the resulting model does not
require specific knowledge of the individual system, is interpretable, and can
be used as a forward, inverse, and energy model. Previously these properties
were only obtained when using system identification techniques that require
knowledge of the kinematic structure. We apply DeLaN to learning dynamics
models and apply these models to control simulated and physical rigid body
systems. The results show that the proposed approach obtains dynamics models
that can be applied to physical systems for real-time control. Compared to
standard deep networks, the physics-inspired models learn better models and
capture the underlying structure of the dynamics.

negative:
Deep neural networks have introduced novel and useful tools to the machine
learning community. Other types of classifiers can potentially make use of
these tools as well to improve their performance and generality. This paper
reviews the current state of the art for deep learning classifier technologies
that are being used outside of deep neural networks. Non-network classifiers
can employ many components found in deep neural network architectures. In this
paper, we review the feature learning, optimization, and regularization methods
that form a core of deep network technologies. We then survey non-neural
network learning algorithms that make innovative use of these methods to
improve classification. Because many opportunities and challenges still exist,
we discuss directions that can be pursued to expand the area of deep learning
for a variety of classification algorithms.

negative:
Embeddings are a fundamental component of many modern machine learning and
natural language processing models. Understanding them and visualizing them is
essential for gathering insights about the information they capture and the
behavior of the models. State of the art in analyzing embeddings consists in
projecting them in two-dimensional planes without any interpretable semantics
associated to the axes of the projection, which makes detailed analyses and
comparison among multiple sets of embeddings challenging. In this work, we
propose to use explicit axes defined as algebraic formulae over embeddings to
project them into a lower dimensional, but semantically meaningful subspace, as
a simple yet effective analysis and visualization methodology. This methodology
assigns an interpretable semantics to the measures of variability and the axes
of visualizations, allowing for both comparisons among different sets of
embeddings and fine-grained inspection of the embedding spaces. We demonstrate
the power of the proposed methodology through a series of case studies that
make use of visualizations constructed around the underlying methodology and
through a user study. The results show how the methodology is effective at
providing more profound insights than classical projection methods and how it
is widely applicable to many other use cases.

negative:
Despite the remarkable success of deep learning in pattern recognition, deep
network models face the problem of training a large number of parameters. In
this paper, we propose and evaluate a novel multi-path wavelet neural network
architecture for image classification with far less number of trainable
parameters. The model architecture consists of a multi-path layout with several
levels of wavelet decompositions performed in parallel followed by fully
connected layers. These decomposition operations comprise wavelet neurons with
learnable parameters, which are updated during the training phase using the
back-propagation algorithm. We evaluate the performance of the introduced
network using common image datasets without data augmentation except for SVHN
and compare the results with influential deep learning models. Our findings
support the possibility of reducing the number of parameters significantly in
deep neural networks without compromising its accuracy.

negative:
The lack of interpretability in current deep learning models causes serious
concerns as they are extensively used for various life-critical applications.
Hence, it is of paramount importance to develop interpretable deep learning
models. In this paper, we consider the problem of blind deconvolution and
propose a novel model-aware deep architecture that allows for the recovery of
both the blur kernel and the sharp image from the blurred image. In particular,
we propose the Deep Unfolded Richardson-Lucy (Deep-URL) framework -- an
interpretable deep-learning architecture that can be seen as an amalgamation of
classical estimation technique and deep neural network, and consequently leads
to improved performance. Our numerical investigations demonstrate significant
improvement compared to state-of-the-art algorithms.

negative:
Deep Learning is a state-of-the-art technique to make inference on extensive
or complex data. As a black box model due to their multilayer nonlinear
structure, Deep Neural Networks are often criticized to be non-transparent and
their predictions not traceable by humans. Furthermore, the models learn from
artificial datasets, often with bias or contaminated discriminating content.
Through their increased distribution, decision-making algorithms can contribute
promoting prejudge and unfairness which is not easy to notice due to lack of
transparency. Hence, scientists developed several so-called explanators or
explainers which try to point out the connection between input and output to
represent in a simplified way the inner structure of machine learning black
boxes. In this survey we differ the mechanisms and properties of explaining
systems for Deep Neural Networks for Computer Vision tasks. We give a
comprehensive overview about taxonomy of related studies and compare several
survey papers that deal with explainability in general. We work out the
drawbacks and gaps and summarize further research ideas.

negative:
In recent years, the use of sophisticated statistical models that influence
decisions in domains of high societal relevance is on the rise. Although these
models can often bring substantial improvements in the accuracy and efficiency
of organizations, many governments, institutions, and companies are reluctant
to their adoption as their output is often difficult to explain in
human-interpretable ways. Hence, these models are often regarded as
black-boxes, in the sense that their internal mechanisms can be opaque to human
audit. In real-world applications, particularly in domains where decisions can
have a sensitive impact--e.g., criminal justice, estimating credit scores,
insurance risk, health risks, etc.--model interpretability is desired.
Recently, the academic literature has proposed a substantial amount of methods
for providing interpretable explanations to machine learning models. This
survey reviews the most relevant and novel methods that form the
state-of-the-art for addressing the particular problem of explaining individual
instances in machine learning. It seeks to provide a succinct review that can
guide data science and machine learning practitioners in the search for
appropriate methods to their problem domain.

negative:
Interpretability of deep neural networks (DNNs) is essential since it enables
users to understand the overall strengths and weaknesses of the models, conveys
an understanding of how the models will behave in the future, and how to
diagnose and correct potential problems. However, it is challenging to reason
about what a DNN actually does due to its opaque or black-box nature. To
address this issue, we propose a novel technique to improve the
interpretability of DNNs by leveraging the rich semantic information embedded
in human descriptions. By concentrating on the video captioning task, we first
extract a set of semantically meaningful topics from the human descriptions
that cover a wide range of visual concepts, and integrate them into the model
with an interpretive loss. We then propose a prediction difference maximization
algorithm to interpret the learned features of each neuron. Experimental
results demonstrate its effectiveness in video captioning using the
interpretable features, which can also be transferred to video action
recognition. By clearly understanding the learned features, users can easily
revise false predictions via a human-in-the-loop procedure.

negative:
Machine learning models that first learn a representation of a domain in
terms of human-understandable concepts, then use it to make predictions, have
been proposed to facilitate interpretation and interaction with models trained
on high-dimensional data. However these methods have important limitations: the
way they define concepts are not inherently interpretable, and they assume that
concept labels either exist for individual instances or can easily be acquired
from users. These limitations are particularly acute for high-dimensional
tabular features. We propose an approach for learning a set of transparent
concept definitions in high-dimensional tabular data that relies on users
labeling concept features instead of individual instances. Our method produces
concepts that both align with users' intuitive sense of what a concept means,
and facilitate prediction of the downstream label by a transparent machine
learning model. This ensures that the full model is transparent and intuitive,
and as predictive as possible given this constraint. We demonstrate with
simulated user feedback on real prediction problems, including one in a
clinical domain, that this kind of direct feedback is much more efficient at
learning solutions that align with ground truth concept definitions than
alternative transparent approaches that rely on labeling instances or other
existing interaction mechanisms, while maintaining similar predictive
performance.

negative:
The decision-making process of many state-of-the-art machine learning models
is inherently inscrutable to the extent that it is impossible for a human to
interpret the model directly: they are black box models. This has led to a call
for research on explaining black box models, for which there are two main
approaches. Global explanations that aim to explain a model's decision making
process in general, and local explanations that aim to explain a single
prediction. Since it remains challenging to establish fidelity to black box
models in globally interpretable approximations, much attention is put on local
explanations. However, whether local explanations are able to reliably
represent the black box model and provide useful insights remains an open
question. We present Global Aggregations of Local Explanations (GALE) with the
objective to provide insights in a model's global decision making process.
Overall, our results reveal that the choice of aggregation matters. We find
that the global importance introduced by Local Interpretable Model-agnostic
Explanations (LIME) does not reliably represent the model's global behavior.
Our proposed aggregations are better able to represent how features affect the
model's predictions, and to provide global insights by identifying
distinguishing features.

negative:
Black-box risk scoring models permeate our lives, yet are typically
proprietary or opaque. We propose Distill-and-Compare, a model distillation and
comparison approach to audit such models. To gain insight into black-box
models, we treat them as teachers, training transparent student models to mimic
the risk scores assigned by black-box models. We compare the student model
trained with distillation to a second un-distilled transparent model trained on
ground-truth outcomes, and use differences between the two models to gain
insight into the black-box model. Our approach can be applied in a realistic
setting, without probing the black-box model API. We demonstrate the approach
on four public data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending
Club. We also propose a statistical test to determine if a data set is missing
key features used to train the black-box model. Our test finds that the
ProPublica data is likely missing key feature(s) used in COMPAS.

negative:
Fisher information and natural gradient provided deep insights and powerful
tools to artificial neural networks. However related analysis becomes more and
more difficult as the learner's structure turns large and complex. This paper
makes a preliminary step towards a new direction. We extract a local component
of a large neuron system, and defines its relative Fisher information metric
that describes accurately this small component, and is invariant to the other
parts of the system. This concept is important because the geometry structure
is much simplified and it can be easily applied to guide the learning of neural
networks. We provide an analysis on a list of commonly used components, and
demonstrate how to use this concept to further improve optimization.

negative:
Integrating physics models within machine learning models holds considerable
promise toward learning robust models with improved interpretability and
abilities to extrapolate. In this work, we focus on the integration of
incomplete physics models into deep generative models. In particular, we
introduce an architecture of variational autoencoders (VAEs) in which a part of
the latent space is grounded by physics. A key technical challenge is to strike
a balance between the incomplete physics and trainable components such as
neural networks for ensuring that the physics part is used in a meaningful
manner. To this end, we propose a regularized learning method that controls the
effect of the trainable components and preserves the semantics of the
physics-based latent variables as intended. We not only demonstrate generative
performance improvements over a set of synthetic and real-world datasets, but
we also show that we learn robust models that can consistently extrapolate
beyond the training distribution in a meaningful manner. Moreover, we show that
we can control the generative process in an interpretable manner.

negative:
The methods of extracting image features are the key to many image processing
tasks. At present, the most popular method is the deep neural network which can
automatically extract robust features through end-to-end training instead of
hand-crafted feature extraction. However, the deep neural network currently
faces many challenges: 1) its effectiveness is heavily dependent on large
datasets, so the computational complexity is very high; 2) it is usually
regarded as a black box model with poor interpretability. To meet the above
challenges, a more interpretable and scalable feature learning method, i.e.,
deep image feature learning with fuzzy rules (DIFL-FR), is proposed in the
paper, which combines the rule-based fuzzy modeling technique and the deep
stacked learning strategy. The method progressively learns image features
through a layer-by-layer manner based on fuzzy rules, so the feature learning
process can be better explained by the generated rules. More importantly, the
learning process of the method is only based on forward propagation without
back propagation and iterative learning, which results in the high learning
efficiency. In addition, the method is under the settings of unsupervised
learning and can be easily extended to scenes of supervised and semi-supervised
learning. Extensive experiments are conducted on image datasets of different
scales. The results obviously show the effectiveness of the proposed method.

negative:
Interpretable deep learning is a fundamental building block towards safer AI,
especially when the deployment possibilities of deep learning-based
computer-aided medical diagnostic systems are so eminent. However, without a
computational formulation of black-box interpretation, general interpretability
research rely heavily on subjective bias. Clear decision structure of the
medical diagnostics lets us approximate the decision process of a radiologist
as a model - removed from subjective bias. We define the process of
interpretation as a finite communication between a known model and a black-box
model to optimally map the black box's decision process in the known model.
Consequently, we define interpretability as maximal information gain over the
initial uncertainty about the black-box's decision within finite communication.
We relax this definition based on the observation that diagnostic
interpretation is typically achieved by a process of minimal querying. We
derive an algorithm to calculate diagnostic interpretability. The usual
question of accuracy-interpretability tradeoff, i.e. whether a black-box
model's prediction accuracy is dependent on its ability to be interpreted by a
known source model, does not arise in this theory. With multiple example
simulation experiments of various complexity levels, we demonstrate the working
of such a theoretical model in synthetic supervised classification scenarios.

negative:
While deep learning has resulted in major breakthroughs in many application
domains, the frameworks commonly used in deep learning remain fragile to
artificially-crafted and imperceptible changes in the data. In response to this
fragility, adversarial training has emerged as a principled approach for
enhancing the robustness of deep learning with respect to norm-bounded
perturbations. However, there are other sources of fragility for deep learning
that are arguably more common and less thoroughly studied. Indeed, natural
variation such as lighting or weather conditions can significantly degrade the
accuracy of trained neural networks, proving that such natural variation
presents a significant challenge for deep learning.
  In this paper, we propose a paradigm shift from perturbation-based
adversarial robustness toward model-based robust deep learning. Our objective
is to provide general training algorithms that can be used to train deep neural
networks to be robust against natural variation in data. Critical to our
paradigm is first obtaining a model of natural variation which can be used to
vary data over a range of natural conditions. Such models may be either known a
priori or else learned from data. In the latter case, we show that deep
generative models can be used to learn models of natural variation that are
consistent with realistic conditions. We then exploit such models in three
novel model-based robust training algorithms in order to enhance the robustness
of deep learning with respect to the given model. Our extensive experiments
show that across a variety of naturally-occurring conditions and across various
datasets, deep neural networks trained with our model-based algorithms
significantly outperform both standard deep learning algorithms as well as
norm-bounded robust deep learning algorithms.

negative:
A new ensemble framework for interpretable model called Linear Iterative
Feature Embedding (LIFE) has been developed to achieve high prediction
accuracy, easy interpretation and efficient computation simultaneously. The
LIFE algorithm is able to fit a wide single-hidden-layer neural network (NN)
accurately with three steps: defining the subsets of a dataset by the linear
projections of neural nodes, creating the features from multiple narrow
single-hidden-layer NNs trained on the different subsets of the data, combining
the features with a linear model. The theoretical rationale behind LIFE is also
provided by the connection to the loss ambiguity decomposition of stack
ensemble methods. Both simulation and empirical experiments confirm that LIFE
consistently outperforms directly trained single-hidden-layer NNs and also
outperforms many other benchmark models, including multi-layers Feed Forward
Neural Network (FFNN), Xgboost, and Random Forest (RF) in many experiments. As
a wide single-hidden-layer NN, LIFE is intrinsically interpretable. Meanwhile,
both variable importance and global main and interaction effects can be easily
created and visualized. In addition, the parallel nature of the base learner
building makes LIFE computationally efficient by leveraging parallel computing.

negative:
Machine learning models are becoming increasingly popular in different types
of settings. This is mainly caused by their ability to achieve a level of
predictive performance that is hard to match by human experts in this new era
of big data. With this usage growth comes an increase of the requirements for
accountability and understanding of the models' predictions. However, the
degree of sophistication of the most successful models (e.g. ensembles, deep
learning) is becoming a large obstacle to this endeavour as these models are
essentially black boxes. In this paper we describe two general approaches that
can be used to provide interpretable descriptions of the expected performance
of any black box classification model. These approaches are of high practical
relevance as they provide means to uncover and describe in an interpretable way
situations where the models are expected to have a performance that deviates
significantly from their average behaviour. This may be of critical relevance
for applications where costly decisions are driven by the predictions of the
models, as it can be used to warn end users against the usage of the models in
some specific cases.

negative:
There is a rich and growing literature on producing local
contrastive/counterfactual explanations for black-box models (e.g. neural
networks).
  In these methods, for an input, an explanation is in the form of a contrast
point differing in very few features from the original input and lying in a
different class. Other works try to build globally interpretable models like
decision trees and rule lists based on the data using actual labels or based on
the black-box models predictions. Although these interpretable global models
can be useful, they may not be consistent with local explanations from a
specific black-box of choice. In this work, we explore the question: Can we
produce a transparent global model that is simultaneously accurate and
consistent with the local (contrastive) explanations of the black-box model? We
introduce a natural local consistency metric that quantifies if the local
explanations and predictions of the black-box model are also consistent with
the proxy global transparent model. Based on a key insight we propose a novel
method where we create custom boolean features from sparse local contrastive
explanations of the black-box model and then train a globally transparent model
on just these, and showcase empirically that such models have higher local
consistency compared with other known strategies, while still being close in
performance to models that are trained with access to the original data.

negative:
Deep neural networks have frequently been used to directly learn
representations useful for a given task from raw input data. In terms of
overall performance metrics, machine learning solutions employing deep
representations frequently have been reported to greatly outperform those using
hand-crafted feature representations. At the same time, they may pick up on
aspects that are predominant in the data, yet not actually meaningful or
interpretable. In this paper, we therefore propose a systematic way to test the
trustworthiness of deep music representations, considering musical semantics.
The underlying assumption is that in case a deep representation is to be
trusted, distance consistency between known related points should be maintained
both in the input audio space and corresponding latent deep space. We generate
known related points through semantically meaningful transformations, both
considering imperceptible and graver transformations. Then, we examine within-
and between-space distance consistencies, both considering audio space and
latent embedded space, the latter either being a result of a conventional
feature extractor or a deep encoder. We illustrate how our method, as a
complement to task-specific performance, provides interpretable insight into
what a network may have captured from training data signals.

negative:
Training deep learning models on mobile devices recently becomes possible,
because of increasing computation power on mobile hardware and the advantages
of enabling high user experiences. Most of the existing work on machine
learning at mobile devices is focused on the inference of deep learning models
(particularly convolutional neural network and recurrent neural network), but
not training. The performance characterization of training deep learning models
on mobile devices is largely unexplored, although understanding the performance
characterization is critical for designing and implementing deep learning
models on mobile devices.
  In this paper, we perform a variety of experiments on a representative mobile
device (the NVIDIA TX2) to study the performance of training deep learning
models. We introduce a benchmark suite and tools to study performance of
training deep learning models on mobile devices, from the perspectives of
memory consumption, hardware utilization, and power consumption. The tools can
correlate performance results with fine-grained operations in deep learning
models, providing capabilities to capture performance variance and problems at
a fine granularity. We reveal interesting performance problems and
opportunities, including under-utilization of heterogeneous hardware, large
energy consumption of the memory, and high predictability of workload
characterization. Based on the performance analysis, we suggest interesting
research directions.

negative:
Deep neural networks work well at approximating complicated functions when
provided with data and trained by gradient descent methods. At the same time,
there is a vast amount of existing functions that programmatically solve
different tasks in a precise manner eliminating the need for training. In many
cases, it is possible to decompose a task to a series of functions, of which
for some we may prefer to use a neural network to learn the functionality,
while for others the preferred method would be to use existing black-box
functions. We propose a method for end-to-end training of a base neural network
that integrates calls to existing black-box functions. We do so by
approximating the black-box functionality with a differentiable neural network
in a way that drives the base network to comply with the black-box function
interface during the end-to-end optimization process. At inference time, we
replace the differentiable estimator with its external black-box
non-differentiable counterpart such that the base network output matches the
input arguments of the black-box function. Using this "Estimate and Replace"
paradigm, we train a neural network, end to end, to compute the input to
black-box functionality while eliminating the need for intermediate labels. We
show that by leveraging the existing precise black-box function during
inference, the integrated model generalizes better than a fully differentiable
model, and learns more efficiently compared to RL-based methods.

negative:
Being able to interpret, or explain, the predictions made by a machine
learning model is of fundamental importance. This is especially true when there
is interest in deploying data-driven models to make high-stakes decisions, e.g.
in healthcare. While recent years have seen an increasing interest in
interpretable machine learning research, this field is currently lacking an
agreed-upon definition of interpretability, and some researchers have called
for a more active conversation towards a rigorous approach to interpretability.
Joining this conversation, we claim in this paper that the difficulty of
interpreting a complex model stems from the existing interactions among
features. We argue that by enforcing monotonicity between features and outputs,
we are able to reason about the effect of a single feature on an output
independently from other features, and consequently better understand the
model. We show how to structurally introduce this constraint in deep learning
models by adding new simple layers. We validate our model on benchmark
datasets, and compare our results with previously proposed interpretable
models.

negative:
Black box machine learning models are currently being used for high stakes
decision-making throughout society, causing problems throughout healthcare,
criminal justice, and in other domains. People have hoped that creating methods
for explaining these black box models will alleviate some of these problems,
but trying to \textit{explain} black box models, rather than creating models
that are \textit{interpretable} in the first place, is likely to perpetuate bad
practices and can potentially cause catastrophic harm to society. There is a
way forward -- it is to design models that are inherently interpretable. This
manuscript clarifies the chasm between explaining black boxes and using
inherently interpretable models, outlines several key reasons why explainable
black boxes should be avoided in high-stakes decisions, identifies challenges
to interpretable machine learning, and provides several example applications
where interpretable models could potentially replace black box models in
criminal justice, healthcare, and computer vision.

negative:
Classical learning theory suggests that the optimal generalization
performance of a machine learning model should occur at an intermediate model
complexity, with simpler models exhibiting high bias and more complex models
exhibiting high variance of the predictive function. However, such a simple
trade-off does not adequately describe deep learning models that simultaneously
attain low bias and variance in the heavily overparameterized regime. A primary
obstacle in explaining this behavior is that deep learning algorithms typically
involve multiple sources of randomness whose individual contributions are not
visible in the total variance. To enable fine-grained analysis, we describe an
interpretable, symmetric decomposition of the variance into terms associated
with the randomness from sampling, initialization, and the labels. Moreover, we
compute the high-dimensional asymptotic behavior of this decomposition for
random feature kernel regression, and analyze the strikingly rich phenomenology
that arises. We find that the bias decreases monotonically with the network
width, but the variance terms exhibit non-monotonic behavior and can diverge at
the interpolation boundary, even in the absence of label noise. The divergence
is caused by the \emph{interaction} between sampling and initialization and can
therefore be eliminated by marginalizing over samples (i.e. bagging) \emph{or}
over the initial parameters (i.e. ensemble learning).

negative:
In the last years many accurate decision support systems have been
constructed as black boxes, that is as systems that hide their internal logic
to the user. This lack of explanation constitutes both a practical and an
ethical issue. The literature reports many approaches aimed at overcoming this
crucial weakness sometimes at the cost of scarifying accuracy for
interpretability. The applications in which black box decision systems can be
used are various, and each approach is typically developed to provide a
solution for a specific problem and, as a consequence, delineating explicitly
or implicitly its own definition of interpretability and explanation. The aim
of this paper is to provide a classification of the main problems addressed in
the literature with respect to the notion of explanation and the type of black
box system. Given a problem definition, a black box type, and a desired
explanation this survey should help the researcher to find the proposals more
useful for his own work. The proposed classification of approaches to open
black box models should also be useful for putting the many research open
questions in perspective.

negative:
Deep Learning based models are currently dominating most state-of-the-art
solutions for disease prediction. Existing works employ RNNs along with
multiple levels of attention mechanisms to provide interpretability. These deep
learning models, with trainable parameters running into millions, require huge
amounts of compute and data to train and deploy. These requirements are
sometimes so huge that they render usage of such models as unfeasible. We
address these challenges by developing a simpler yet interpretable non-deep
learning based model for application to EHR data. We model and showcase our
work's results on the task of predicting first occurrence of a diagnosis, often
overlooked in existing works. We push the capabilities of a tree based model
and come up with a strong baseline for more sophisticated models. Its
performance shows an improvement over deep learning based solutions (both, with
and without the first-occurrence constraint) all the while maintaining
interpretability.

negative:
Deep ensembles excel in large-scale image classification tasks both in terms
of prediction accuracy and calibration. Despite being simple to train, the
computation and memory cost of deep ensembles limits their practicability.
While some recent works propose to distill an ensemble model into a single
model to reduce such costs, there is still a performance gap between the
ensemble and distilled models. We propose a simple approach for reducing this
gap, i.e., making the distilled performance close to the full ensemble. Our key
assumption is that a distilled model should absorb as much function diversity
inside the ensemble as possible. We first empirically show that the typical
distillation procedure does not effectively transfer such diversity, especially
for complex models that achieve near-zero training error. To fix this, we
propose a perturbation strategy for distillation that reveals diversity by
seeking inputs for which ensemble member outputs disagree. We empirically show
that a model distilled with such perturbed samples indeed exhibits enhanced
diversity, leading to improved performance.

negative:
It is known that unsupervised nonlinear dimensionality reduction and
clustering is sensitive to the selection of hyperparameters, particularly for
deep learning based methods, which hinders its practical use. How to select a
proper network structure that may be dramatically different in different
applications is a hard issue for deep models, given little prior knowledge of
data. In this paper, we aim to automatically determine the optimal network
structure of a deep model, named multilayer bootstrap networks (MBN), via
simple ensemble learning and selection techniques. Specifically, we first
propose an MBN ensemble (MBN-E) algorithm which concatenates the sparse outputs
of a set of MBN base models with different network structures into a new
representation. Then, we take the new representation produced by MBN-E as a
reference for selecting the optimal MBN base models. Moreover, we propose a
fast version of MBN-E (fMBN-E), which is not only theoretically even faster
than a single standard MBN but also does not increase the estimation error of
MBN-E. Importantly, MBN-E and its ensemble selection techniques maintain the
simple formulation of MBN that is based on one-nearest-neighbor learning.
Empirically, comparing to a number of advanced deep clustering methods and as
many as 20 representative unsupervised ensemble learning and selection methods,
the proposed methods reach the state-of-the-art performance without manual
hyperparameter tuning. fMBN-E is empirically even hundreds of times faster than
MBN-E without suffering performance degradation. The applications to image
segmentation and graph data mining further demonstrate the advantage of the
proposed methods.

negative:
We study the task of replicating the functionality of black-box neural
models, for which we only know the output class probabilities provided for a
set of input images. We assume back-propagation through the black-box model is
not possible and its training images are not available, e.g. the model could be
exposed only through an API. In this context, we present a teacher-student
framework that can distill the black-box (teacher) model into a student model
with minimal accuracy loss. To generate useful data samples for training the
student, our framework (i) learns to generate images on a proxy data set (with
images and classes different from those used to train the black-box) and (ii)
applies an evolutionary strategy to make sure that each generated data sample
exhibits a high response for a specific class when given as input to the black
box. Our framework is compared with several baseline and state-of-the-art
methods on three benchmark data sets. The empirical evidence indicates that our
model is superior to the considered baselines. Although our method does not
back-propagate through the black-box network, it generally surpasses
state-of-the-art methods that regard the teacher as a glass-box model. Our code
is available at: https://github.com/antoniobarbalau/black-box-ripper.

negative:
Deep learning has achieved a great success in many areas, from computer
vision to natural language processing, to game playing, and much more. Yet,
what deep learning is really doing is still an open question. There are a lot
of works in this direction. For example, [5] tried to explain deep learning by
group renormalization, and [6] tried to explain deep learning from the view of
functional approximation. In order to address this very crucial question, here
we see deep learning from perspective of mechanical learning and learning
machine (see [1], [2]). From this particular angle, we can see deep learning
much better and answer with confidence: What deep learning is really doing? why
it works well, how it works, and how much data is necessary for learning. We
also will discuss advantages and disadvantages of deep learning at the end of
this work.

negative:
Learning, taking into account full distribution of the data, referred to as
generative, is not feasible with deep neural networks (DNNs) because they model
only the conditional distribution of the outputs given the inputs. Current
solutions are either based on joint probability models facing difficult
estimation problems or learn two separate networks, mapping inputs to outputs
(recognition) and vice-versa (generation). We propose an intermediate approach.
First, we show that forward computation in DNNs with logistic sigmoid
activations corresponds to a simplified approximate Bayesian inference in a
directed probabilistic multi-layer model. This connection allows to interpret
DNN as a probabilistic model of the output and all hidden units given the
input. Second, we propose that in order for the recognition and generation
networks to be more consistent with the joint model of the data, weights of the
recognition and generator network should be related by transposition. We
demonstrate in a tentative experiment that such a coupled pair can be learned
generatively, modelling the full distribution of the data, and has enough
capacity to perform well in both recognition and generation.

negative:
Deep learning has recently become very popular on account of its incredible
success in many complex data-driven applications, such as image classification
and speech recognition. The database community has worked on data-driven
applications for many years, and therefore should be playing a lead role in
supporting this new wave. However, databases and deep learning are different in
terms of both techniques and applications. In this paper, we discuss research
problems at the intersection of the two fields. In particular, we discuss
possible improvements for deep learning systems from a database perspective,
and analyze database applications that may benefit from deep learning
techniques.

negative:
Deep neural networks are able to solve tasks across a variety of domains and
modalities of data. Despite many empirical successes, we lack the ability to
clearly understand and interpret the learned internal mechanisms that
contribute to such effective behaviors or, more critically, failure modes. In
this work, we present a general method for visualizing an arbitrary neural
network's inner mechanisms and their power and limitations. Our dataset-centric
method produces visualizations of how a trained network attends to components
of its inputs. The computed "attention masks" support improved interpretability
by highlighting which input attributes are critical in determining output. We
demonstrate the effectiveness of our framework on a variety of deep neural
network architectures in domains from computer vision, natural language
processing, and reinforcement learning. The primary contribution of our
approach is an interpretable visualization of attention that provides unique
insights into the network's underlying decision-making process irrespective of
the data modality.

negative:
With machine learning models being increasingly used to aid decision making
even in high-stakes domains, there has been a growing interest in developing
interpretable models. Although many supposedly interpretable models have been
proposed, there have been relatively few experimental studies investigating
whether these models achieve their intended effects, such as making people more
closely follow a model's predictions when it is beneficial for them to do so or
enabling them to detect when a model has made a mistake. We present a sequence
of pre-registered experiments (N=3,800) in which we showed participants
functionally identical models that varied only in two factors commonly thought
to make machine learning models more or less interpretable: the number of
features and the transparency of the model (i.e., whether the model internals
are clear or black box). Predictably, participants who saw a clear model with
few features could better simulate the model's predictions. However, we did not
find that participants more closely followed its predictions. Furthermore,
showing participants a clear model meant that they were less able to detect and
correct for the model's sizable mistakes, seemingly due to information
overload. These counterintuitive findings emphasize the importance of testing
over intuition when developing interpretable models.

negative:
We consider the task of solving generic inverse problems, where one wishes to
determine the hidden parameters of a natural system that will give rise to a
particular set of measurements. Recently many new approaches based upon deep
learning have arisen generating impressive results. We conceptualize these
models as different schemes for efficiently, but randomly, exploring the space
of possible inverse solutions. As a result, the accuracy of each approach
should be evaluated as a function of time rather than a single estimated
solution, as is often done now. Using this metric, we compare several
state-of-the-art inverse modeling approaches on four benchmark tasks: two
existing tasks, one simple task for visualization and one new task from
metamaterial design. Finally, inspired by our conception of the inverse
problem, we explore a solution that uses a deep learning model to approximate
the forward model, and then uses backpropagation to search for good inverse
solutions. This approach, termed the neural-adjoint, achieves the best
performance in many scenarios.

negative:
Kernel methods are powerful machine learning techniques which implement
generic non-linear functions to solve complex tasks in a simple way. They Have
a solid mathematical background and exhibit excellent performance in practice.
However, kernel machines are still considered black-box models as the feature
mapping is not directly accessible and difficult to interpret.The aim of this
work is to show that it is indeed possible to interpret the functions learned
by various kernel methods is intuitive despite their complexity. Specifically,
we show that derivatives of these functions have a simple mathematical
formulation, are easy to compute, and can be applied to many different
problems. We note that model function derivatives in kernel machines is
proportional to the kernel function derivative. We provide the explicit
analytic form of the first and second derivatives of the most common kernel
functions with regard to the inputs as well as generic formulas to compute
higher order derivatives. We use them to analyze the most used supervised and
unsupervised kernel learning methods: Gaussian Processes for regression,
Support Vector Machines for classification, Kernel Entropy Component Analysis
for density estimation, and the Hilbert-Schmidt Independence Criterion for
estimating the dependency between random variables. For all cases we expressed
the derivative of the learned function as a linear combination of the kernel
function derivative. Moreover we provide intuitive explanations through
illustrative toy examples and show how to improve the interpretation of real
applications in the context of spatiotemporal Earth system data cubes. This
work reflects on the observation that function derivatives may play a crucial
role in kernel methods analysis and understanding.

negative:
Due to the dominant position of deep learning (mostly deep neural networks)
in various artificial intelligence applications, recently, ensemble learning
based on deep neural networks (ensemble deep learning) has shown significant
performances in improving the generalization of learning system. However, since
modern deep neural networks usually have millions to billions of parameters,
the time and space overheads for training multiple base deep learners and
testing with the ensemble deep learner are far greater than that of traditional
ensemble learning. Though several algorithms of fast ensemble deep learning
have been proposed to promote the deployment of ensemble deep learning in some
applications, further advances still need to be made for many applications in
specific fields, where the developing time and computing resources are usually
restricted or the data to be processed is of large dimensionality. An urgent
problem needs to be solved is how to take the significant advantages of
ensemble deep learning while reduce the required expenses so that many more
applications in specific fields can benefit from it. For the alleviation of
this problem, it is essential to know about how ensemble learning has developed
under the era of deep learning. Thus, in this article, we present fundamental
discussions focusing on data analyses of published works, methodologies, recent
advances and unattainability of traditional ensemble learning and ensemble deep
learning. We hope this article will be helpful to realize the intrinsic
problems and technical challenges faced by future developments of ensemble
learning under the era of deep learning.

negative:
Should we input known genome sequence features or input sequence itself in
deep learning framework? As deep learning more popular in various applications,
researchers often come to question whether to generate features or use raw
sequences for deep learning. To answer this question, we study the prediction
accuracy of precursor miRNA prediction of feature-based deep belief network and
sequence-based convolution neural network. Tested on a variant of six-layer
convolution neural net and three-layer deep belief network, we find the raw
sequence input based convolution neural network model performs similar or
slightly better than feature based deep belief networks with best accuracy
values of 0.995 and 0.990, respectively. Both the models outperform existing
benchmarks models. The results shows us that if provided large enough data,
well devised raw sequence based deep learning models can replace feature based
deep learning models. However, construction of well behaved deep learning model
can be very challenging. In cased features can be easily extracted,
feature-based deep learning models may be a better alternative.

negative:
Advancements in genomic research such as high-throughput sequencing
techniques have driven modern genomic studies into "big data" disciplines. This
data explosion is constantly challenging conventional methods used in genomics.
In parallel with the urgent demand for robust algorithms, deep learning has
succeeded in a variety of fields such as vision, speech, and text processing.
Yet genomics entails unique challenges to deep learning since we are expecting
from deep learning a superhuman intelligence that explores beyond our knowledge
to interpret the genome. A powerful deep learning model should rely on
insightful utilization of task-specific knowledge. In this paper, we briefly
discuss the strengths of different deep learning models from a genomic
perspective so as to fit each particular task with a proper deep architecture,
and remark on practical considerations of developing modern deep learning
architectures for genomics. We also provide a concise review of deep learning
applications in various aspects of genomic research, as well as pointing out
potential opportunities and obstacles for future genomics applications.

negative:
Many end-user evaluations of data visualization techniques have been run
during the last decades. Their results are cornerstones to build efficient
visualization systems. However, designing such an evaluation is always complex
and time-consuming and may end in a lack of statistical evidence and
reproducibility. We believe that modern and efficient computer vision
techniques, such as deep convolutional neural networks (CNNs), may help
visualization researchers to build and/or adjust their evaluation hypothesis.
The basis of our idea is to train machine learning models on several
visualization techniques to solve a specific task. Our assumption is that it is
possible to compare the efficiency of visualization techniques based on the
performance of their corresponding model. As current machine learning models
are not able to strictly reflect human capabilities, including their
imperfections, such results should be interpreted with caution. However, we
think that using machine learning-based pre-evaluation, as a pre-process of
standard user evaluations, should help researchers to perform a more exhaustive
study of their design space. Thus, it should improve their final user
evaluation by providing it better test cases. In this paper, we present the
results of two experiments we have conducted to assess how correlated the
performance of users and computer vision techniques can be. That study compares
two mainstream graph visualization techniques: node-link (\NL) and
adjacency-matrix (\MD) diagrams. Using two well-known deep convolutional neural
networks, we partially reproduced user evaluations from Ghoniem \textit{et al.}
and from Okoe \textit{et al.}. These experiments showed that some user
evaluation results can be reproduced automatically.

negative:
Understanding how a learned black box works is of crucial interest for the
future of Machine Learning. In this paper, we pioneer the question of the
global interpretability of learned black box models that assign numerical
values to symbolic sequential data. To tackle that task, we propose a spectral
algorithm for the extraction of weighted automata (WA) from such black boxes.
This algorithm does not require the access to a dataset or to the inner
representation of the black box: the inferred model can be obtained solely by
querying the black box, feeding it with inputs and analyzing its outputs.
Experiments using Recurrent Neural Networks (RNN) trained on a wide collection
of 48 synthetic datasets and 2 real datasets show that the obtained
approximation is of great quality.

negative:
My research lies in the intersection of security and machine learning. This
overview summarizes one component of my research: combining computer vision
with malware exploit detection for enhanced security solutions. I will present
the perspectives of efficacy, reliability and resiliency to formulate threat
detection as computer vision problems and develop state-of-the-art image-based
malware classification. Representing malware binary as images provides a direct
visualization of data samples, reduces the efforts for feature extraction, and
consumes the whole binary for holistic structural analysis. Employing transfer
learning of deep neural networks effective for large scale image classification
to malware classification demonstrates superior classification efficacy
compared with classical machine learning algorithms. To enhance reliability of
these vision-based malware detectors, interpretation frameworks can be
constructed on the malware visual representations and useful for extracting
faithful explanation, so that security practitioners have confidence in the
model before deployment. In cyber-security applications, we should always
assume that a malware writer constantly modifies code to bypass detection.
Addressing the resiliency of the malware detectors is equivalently important as
efficacy and reliability. Via understanding the attack surfaces of machine
learning models used for malware detection, we can greatly improve the
robustness of the algorithms to combat malware adversaries in the wild. Finally
I will discuss future research directions worth pursuing in this research
community.

negative:
Deep learning models have achieved huge success in numerous fields, such as
computer vision and natural language processing. However, unlike such fields,
it is hard to apply traditional deep learning models on the graph data due to
the 'node-orderless' property. Normally, adjacency matrices will cast an
artificial and random node-order on the graphs, which renders the performance
of deep models on graph classification tasks extremely erratic, and the
representations learned by such models lack clear interpretability. To
eliminate the unnecessary node-order constraint, we propose a novel model named
Isomorphic Neural Network (IsoNN), which learns the graph representation by
extracting its isomorphic features via the graph matching between input graph
and templates. IsoNN has two main components: graph isomorphic feature
extraction component and classification component. The graph isomorphic feature
extraction component utilizes a set of subgraph templates as the kernel
variables to learn the possible subgraph patterns existing in the input graph
and then computes the isomorphic features. A set of permutation matrices is
used in the component to break the node-order brought by the matrix
representation. Three fully-connected layers are used as the classification
component in IsoNN. Extensive experiments are conducted on benchmark datasets,
the experimental results can demonstrate the effectiveness of ISONN, especially
compared with both classic and state-of-the-art graph classification methods.

negative:
Questions convey information about the questioner, namely what one does not
know. In this paper, we propose a novel approach to allow a learning agent to
ask what it considers as tricky to predict, in the course of producing a final
output. By analyzing when and what it asks, we can make our model more
transparent and interpretable. We first develop this idea to propose a general
framework of deep neural networks that can ask questions, which we call asking
networks. A specific architecture and training process for an asking network is
proposed for the task of colorization, which is an exemplar one-to-many task
and thus a task where asking questions is helpful in performing the task
accurately. Our results show that the model learns to generate meaningful
questions, asks difficult questions first, and utilizes the provided hint more
efficiently than baseline models. We conclude that the proposed asking
framework makes the learning agent reveal its weaknesses, which poses a
promising new direction in developing interpretable and interactive models.

negative:
Deep learning models are widely used for various industrial and scientific
applications. Even though these models have achieved considerable success in
recent years, there exists a lack of understanding of the rationale behind
decisions made by such systems in the machine learning community. This problem
of interpretability is further aggravated by the increasing complexity of such
models. This paper utilizes concepts from machine learning, quantum computation
and quantum field theory to demonstrate how a many valued quantum logic system
naturally arises in a specific class of generative deep learning models called
Convolutional Deep Belief Networks. It provides a robust theoretical framework
for constructing deep learning models equipped with the interpretability of
many valued quantum logic systems without compromising their computing
efficiency.

negative:
To achieve seamless human-robot interactions, robots need to intimately
reason about complex interaction dynamics and future human behaviors within
their motion planning process. However, there is a disconnect between
state-of-the-art neural network-based human behavior models and robot motion
planners -- either the behavior models are limited in their consideration of
downstream planning or a simplified behavior model is used to ensure
tractability of the planning problem. In this work, we present a framework that
fuses together the interpretability and flexibility of trajectory optimization
(TO) with the predictive power of state-of-the-art human trajectory prediction
models. In particular, we leverage gradient information from data-driven
prediction models to explicitly reason about human-robot interaction dynamics
within a gradient-based TO problem. We demonstrate the efficacy of our approach
in a multi-agent scenario whereby a robot is required to safely and efficiently
navigate through a crowd of up to ten pedestrians. We compare against a variety
of planning methods, and show that by explicitly accounting for interaction
dynamics within the planner, our method offers safer and more efficient
behaviors, even yielding proactive and nuanced behaviors such as waiting for a
pedestrian to pass before moving.

negative:
Analog hardware implemented deep learning models are promising for
computation and energy constrained systems such as edge computing devices.
However, the analog nature of the device and the associated many noise sources
will cause changes to the value of the weights in the trained deep learning
models deployed on such devices. In this study, systematic evaluation of the
inference performance of trained popular deep learning models for image
classification deployed on analog devices has been carried out, where additive
white Gaussian noise has been added to the weights of the trained models during
inference. It is observed that deeper models and models with more redundancy in
design such as VGG are more robust to the noise in general. However, the
performance is also affected by the design philosophy of the model, the
detailed structure of the model, the exact machine learning task, as well as
the datasets.

negative:
Understanding generalization in deep learning is arguably one of the most
important questions in deep learning. Deep learning has been successfully
adopted to a large number of problems ranging from pattern recognition to
complex decision making, but many recent researchers have raised many concerns
about deep learning, among which the most important is generalization. Despite
numerous attempts, conventional statistical learning approaches have yet been
able to provide a satisfactory explanation on why deep learning works. A recent
line of works aims to address the problem by trying to predict the
generalization performance through complexity measures. In this competition, we
invite the community to propose complexity measures that can accurately predict
generalization of models. A robust and general complexity measure would
potentially lead to a better understanding of deep learning's underlying
mechanism and behavior of deep models on unseen data, or shed light on better
generalization bounds. All these outcomes will be important for making deep
learning more robust and reliable.

negative:
In recent years, deep neural networks (DNN) have become a highly active area
of research, and shown remarkable achievements on a variety of computer vision
tasks. DNNs, however, are known to often make overconfident yet incorrect
predictions on out-of-distribution samples, which can be a major obstacle to
real-world deployments because the training dataset is always limited compared
to diverse real-world samples. Thus, it is fundamental to provide guarantees of
robustness to the distribution shift between training and test time when we
construct DNN models in practice. Moreover, in many cases, the deep learning
models are deployed as black boxes and the performance has been already
optimized for a training dataset, thus changing the black box itself can lead
to performance degradation. We here study the robustness to the geometric
transformations in a specific condition where the black-box image classifier is
given. We propose an additional learner, \emph{REinforcement Spatial Transform
learner (REST)}, that transforms the warped input data into samples regarded as
in-distribution by the black-box models. Our work aims to improve the
robustness by adding a REST module in front of any black boxes and training
only the REST module without retraining the original black box model in an
end-to-end manner, i.e. we try to convert the real-world data into training
distribution which the performance of the black-box model is best suited for.
We use a confidence score that is obtained from the black-box model to
determine whether the transformed input is drawn from in-distribution. We
empirically show that our method has an advantage in generalization to
geometric transformations and sample efficiency.

negative:
Building surrogate models is one common approach when we attempt to learn
unknown black-box functions. Bayesian optimization provides a framework which
allows us to build surrogate models based on sequential samples drawn from the
function and find the optimum. Tuning algorithmic parameters to optimize the
performance of large, complicated "black-box" application codes is a specific
important application, which aims at finding the optima of black-box functions.
Within the Bayesian optimization framework, the Gaussian process model produces
smooth or continuous sample paths. However, the black-box function in the
tuning problem is often non-smooth. This difficult tuning problem is worsened
by the fact that we usually have limited sequential samples from the black-box
function. Motivated by these issues encountered in tuning, we propose a novel
additive Gaussian process model called clustered Gaussian process (cGP), where
the additive components are induced by clustering. In the examples we studied,
the performance can be improved by as much as 90% among repetitive experiments.
By using this surrogate model, we want to capture the non-smoothness of the
black-box function. In addition to an algorithm for constructing this model, we
also apply the model to several artificial and real applications to evaluate
it.

negative:
Nowadays, the use of machine learning models is becoming a utility in many
applications. Companies deliver pre-trained models encapsulated as application
programming interfaces (APIs) that developers combine with third party
components and their own models and data to create complex data products to
solve specific problems. The complexity of such products and the lack of
control and knowledge of the internals of each component used cause unavoidable
effects, such as lack of transparency, difficulty in auditability, and
emergence of potential uncontrolled risks. They are effectively black-boxes.
Accountability of such solutions is a challenge for the auditors and the
machine learning community. In this work, we propose a wrapper that given a
black-box model enriches its output prediction with a measure of uncertainty.
By using this wrapper, we make the black-box auditable for the accuracy risk
(risk derived from low quality or uncertain decisions) and at the same time we
provide an actionable mechanism to mitigate that risk in the form of decision
rejection; we can choose not to issue a prediction when the risk or uncertainty
in that decision is significant. Based on the resulting uncertainty measure, we
advocate for a rejection system that selects the more confident predictions,
discarding those more uncertain, leading to an improvement in the trustability
of the resulting system. We showcase the proposed technique and methodology in
a practical scenario where a simulated sentiment analysis API based on natural
language processing is applied to different domains. Results demonstrate the
effectiveness of the uncertainty computed by the wrapper and its high
correlation to bad quality predictions and misclassifications.

negative:
Recently, the market on deep learning including not only software but also
hardware is developing rapidly. Big data is collected through IoT devices and
the industry world will analyze them to improve their manufacturing process.
Deep Learning has the hierarchical network architecture to represent the
complicated features of input patterns. Although deep learning can show the
high capability of classification, prediction, and so on, the implementation on
GPU devices are required. We may meet the trade-off between the higher
precision by deep learning and the higher cost with GPU devices. We can success
the knowledge extraction from the trained deep learning with high
classification capability. The knowledge that can realize faster inference of
pre-trained deep network is extracted as IF-THEN rules from the network signal
flow given input data. Some experiment results with benchmark tests for time
series data sets showed the effectiveness of our proposed method related to the
computational speed.

negative:
Machine learning models are increasingly used to automate decisions that
affect humans - deciding who should receive a loan, a job interview, or a
social service. In such applications, a person should have the ability to
change the decision of a model. When a person is denied a loan by a credit
score, for example, they should be able to alter its input variables in a way
that guarantees approval. Otherwise, they will be denied the loan as long as
the model is deployed. More importantly, they will lack the ability to
influence a decision that affects their livelihood.
  In this paper, we frame these issues in terms of recourse, which we define as
the ability of a person to change the decision of a model by altering
actionable input variables (e.g., income vs. age or marital status). We present
integer programming tools to ensure recourse in linear classification problems
without interfering in model development. We demonstrate how our tools can
inform stakeholders through experiments on credit scoring problems. Our results
show that recourse can be significantly affected by standard practices in model
development, and motivate the need to evaluate recourse in practice.

negative:
Many works demonstrate that deep learning system is vulnerable to adversarial
attack. A deep learning system consists of two parts: the deep learning task
and the deep model. Nowadays, most existing works investigate the impact of the
deep model on robustness of deep learning systems, ignoring the impact of the
learning task. In this paper, we adopt the binary and interval label encoding
strategy to redefine the classification task and design corresponding loss to
improve robustness of the deep learning system. Our method can be viewed as
improving the robustness of deep learning systems from both the learning task
and deep model. Experimental results demonstrate that our learning task-aware
method is much more robust than traditional classification while retaining the
accuracy.

negative:
We describe an open-source toolkit for neural machine translation (NMT). The
toolkit prioritizes efficiency, modularity, and extensibility with the goal of
supporting NMT research into model architectures, feature representations, and
source modalities, while maintaining competitive performance and reasonable
training requirements. The toolkit consists of modeling and translation
support, as well as detailed pedagogical documentation about the underlying
techniques.

negative:
Deep neural networks have achieved state of the art accuracy at classifying
molecules with respect to whether they bind to specific protein targets. A key
breakthrough would occur if these models could reveal the fragment
pharmacophores that are causally involved in binding. Extracting chemical
details of binding from the networks could potentially lead to scientific
discoveries about the mechanisms of drug actions. But doing so requires shining
light into the black box that is the trained neural network model, a task that
has proved difficult across many domains. Here we show how the binding
mechanism learned by deep neural network models can be interrogated, using a
recently described attribution method. We first work with carefully constructed
synthetic datasets, in which the 'fragment logic' of binding is fully known. We
find that networks that achieve perfect accuracy on held out test datasets
still learn spurious correlations due to biases in the datasets, and we are
able to exploit this non-robustness to construct adversarial examples that fool
the model. The dataset bias makes these models unreliable for accurately
revealing information about the mechanisms of protein-ligand binding. In light
of our findings, we prescribe a test that checks for dataset bias given a
hypothesis. If the test fails, it indicates that either the model must be
simplified or regularized and/or that the training dataset requires
augmentation.

negative:
Which components of the singular value decomposition of a signal-plus-noise
data matrix are most informative for the inferential task of detecting or
estimating an embedded low-rank signal matrix? Principal component analysis
ascribes greater importance to the components that capture the greatest
variation, i.e., the singular vectors associated with the largest singular
values. This choice is often justified by invoking the Eckart-Young theorem
even though that work addresses the problem of how to best represent a
signal-plus-noise matrix using a low-rank approximation and not how to
best_infer_ the underlying low-rank signal component.
  Here we take a first-principles approach in which we start with a
signal-plus-noise data matrix and show how the spectrum of the noise-only
component governs whether the principal or the middle components of the
singular value decomposition of the data matrix will be the informative
components for inference. Simply put, if the noise spectrum is supported on a
connected interval, in a sense we make precise, then the use of the principal
components is justified. When the noise spectrum is supported on multiple
intervals, then the middle components might be more informative than the
principal components.
  The end result is a proper justification of the use of principal components
in the setting where the noise matrix is i.i.d. Gaussian and the identification
of scenarios, generically involving heterogeneous noise models such as mixtures
of Gaussians, where the middle components might be more informative than the
principal components so that they may be exploited to extract additional
processing gain. Our results show how the blind use of principal components can
lead to suboptimal or even faulty inference because of phase transitions that
separate a regime where the principal components are informative from a regime
where they are uninformative.

negative:
Gene expression profiles have been widely used to characterize patterns of
cellular responses to diseases. As data becomes available, scalable learning
toolkits become essential to processing large datasets using deep learning
models to model complex biological processes. We present an autoencoder to
capture nonlinear relationships recovered from gene expression profiles. The
autoencoder is a nonlinear dimension reduction technique using an artificial
neural network, which learns hidden representations of unlabeled data. We train
the autoencoder on a large collection of tumor samples from the National Cancer
Institute Genomic Data Commons, and obtain a generalized and unsupervised
latent representation. We leverage a HPC-focused deep learning toolkit,
Livermore Big Artificial Neural Network (LBANN) to efficiently parallelize the
training algorithm, reducing computation times from several hours to a few
minutes. With the trained autoencoder, we generate latent representations of a
small dataset, containing pairs of normal and cancer cells of various tumor
types. A novel measure called autoencoder node saliency (ANS) is introduced to
identify the hidden nodes that best differentiate various pairs of cells. We
compare our findings of the best classifying nodes with principal component
analysis and the visualization of t-distributed stochastic neighbor embedding.
We demonstrate that the autoencoder effectively extracts distinct gene features
for multiple learning tasks in the dataset.

negative:
Sparse Principal Component Analysis (sPCA) is a popular matrix factorization
approach based on Principal Component Analysis (PCA) that combines variance
maximization and sparsity with the ultimate goal of improving data
interpretation. When moving from PCA to sPCA, there are a number of
implications that the practitioner needs to be aware of. A relevant one is that
scores and loadings in sPCA may not be orthogonal. For this reason, the
traditional way of computing scores, residuals and variance explained that is
used in the classical PCA cannot directly be applied to sPCA models. This also
affects how sPCA components should be visualized. In this paper we illustrate
this problem both theoretically and numerically using simulations for several
state-of-the-art sPCA algorithms, and provide proper computation of the
different elements mentioned. We show that sPCA approaches present disparate
and limited performance when modeling noise-free, sparse data. In a follow-up
paper, we discuss the theoretical properties that lead to this problem.

negative:
Probabilistic deep learning is deep learning that accounts for uncertainty,
both model uncertainty and data uncertainty. It is based on the use of
probabilistic models and deep neural networks. We distinguish two approaches to
probabilistic deep learning: probabilistic neural networks and deep
probabilistic models. The former employs deep neural networks that utilize
probabilistic layers which can represent and process uncertainty; the latter
uses probabilistic models that incorporate deep neural network components which
capture complex non-linear stochastic relationships between the random
variables. We discuss some major examples of each approach including Bayesian
neural networks and mixture density networks (for probabilistic neural
networks), and variational autoencoders, deep Gaussian processes and deep mixed
effects models (for deep probabilistic models). TensorFlow Probability is a
library for probabilistic modeling and inference which can be used for both
approaches of probabilistic deep learning. We include its code examples for
illustration.

negative:
The thin-file borrowers are customers for whom a creditworthiness assessment
is uncertain due to their lack of credit history; many researchers have used
borrowers' relationships and interactions networks in the form of graphs as an
alternative data source to address this. Incorporating network data is
traditionally made by hand-crafted feature engineering, and lately, the graph
neural network has emerged as an alternative, but it still does not improve
over the traditional method's performance. Here we introduce a framework to
improve credit scoring models by blending several Graph Representation Learning
methods: feature engineering, graph embeddings, and graph neural networks. We
stacked their outputs to produce a single score in this approach. We validated
this framework using a unique multi-source dataset that characterizes the
relationships and credit history for the entire population of a Latin American
country, applying it to credit risk models, application, and behavior,
targeting both individuals and companies.
  Our results show that the graph representation learning methods should be
used as complements, and these should not be seen as self-sufficient methods as
is currently done. In terms of AUC and KS, we enhance the statistical
performance, outperforming traditional methods.
  In Corporate lending, where the gain is much higher, it confirms that
evaluating an unbanked company cannot solely consider its features. The
business ecosystem where these firms interact with their owners, suppliers,
customers, and other companies provides novel knowledge that enables financial
institutions to enhance their creditworthiness assessment.
  Our results let us know when and which group to use graph data and what
effects on performance to expect. They also show the enormous value of graph
data on the unbanked credit scoring problem, principally to help companies'
banking.

negative:
In this paper, we present a statistical-mechanical analysis of deep learning.
We elucidate some of the essential components of deep learning---pre-training
by unsupervised learning and fine tuning by supervised learning. We formulate
the extraction of features from the training data as a margin criterion in a
high-dimensional feature-vector space. The self-organized classifier is then
supplied with small amounts of labelled data, as in deep learning. Although we
employ a simple single-layer perceptron model, rather than directly analyzing a
multi-layer neural network, we find a nontrivial phase transition that is
dependent on the number of unlabelled data in the generalization error of the
resultant classifier. In this sense, we evaluate the efficacy of the
unsupervised learning component of deep learning. The analysis is performed by
the replica method, which is a sophisticated tool in statistical mechanics. We
validate our result in the manner of deep learning, using a simple iterative
algorithm to learn the weight vector on the basis of belief propagation.

negative:
The great success of deep learning shows that its technology contains
profound truth, and understanding its internal mechanism not only has important
implications for the development of its technology and effective application in
various fields, but also provides meaningful insights into the understanding of
human brain mechanism. At present, most of the theoretical research on deep
learning is based on mathematics. This dissertation proposes that the neural
network of deep learning is a physical system, examines deep learning from
three different perspectives: microscopic, macroscopic, and physical world
views, answers multiple theoretical puzzles in deep learning by using physics
principles. For example, from the perspective of quantum mechanics and
statistical physics, this dissertation presents the calculation methods for
convolution calculation, pooling, normalization, and Restricted Boltzmann
Machine, as well as the selection of cost functions, explains why deep learning
must be deep, what characteristics are learned in deep learning, why
Convolutional Neural Networks do not have to be trained layer by layer, and the
limitations of deep learning, etc., and proposes the theoretical direction and
basis for the further development of deep learning now and in the future. The
brilliance of physics flashes in deep learning, we try to establish the deep
learning technology based on the scientific theory of physics.

negative:
With the growing popularity of deep-learning based NLP models, comes a need
for interpretable systems. But what is interpretability, and what constitutes a
high-quality interpretation? In this opinion piece we reflect on the current
state of interpretability evaluation research. We call for more clearly
differentiating between different desired criteria an interpretation should
satisfy, and focus on the faithfulness criteria. We survey the literature with
respect to faithfulness evaluation, and arrange the current approaches around
three assumptions, providing an explicit form to how faithfulness is "defined"
by the community. We provide concrete guidelines on how evaluation of
interpretation methods should and should not be conducted. Finally, we claim
that the current binary definition for faithfulness sets a potentially
unrealistic bar for being considered faithful. We call for discarding the
binary notion of faithfulness in favor of a more graded one, which we believe
will be of greater practical utility.

negative:
Social network analysis is the process of investigating social structures
through the use of networks and graph theory. It combines a variety of
techniques for analyzing the structure of social networks as well as theories
that aim at explaining the underlying dynamics and patterns observed in these
structures. It is an inherently interdisciplinary field which originally
emerged from the fields of social psychology, statistics and graph theory. This
talk will covers the theory of social network analysis, with a short
introduction to graph theory and information spread. Then we will deep dive
into Python code with NetworkX to get a better understanding of the network
components, followed-up by constructing and implying social networks from real
Pandas and textual datasets. Finally we will go over code examples of practical
use-cases such as visualization with matplotlib, social-centrality analysis and
influence maximization for information spread.

negative:
In this study, we define interaction components of different orders between
two input variables based on game theory. We further prove that interaction
components of different orders satisfy several desirable properties.

negative:
Deep neural networks have achieved impressive results on a wide variety of
tasks. However, quantifying uncertainty in the network's output is a
challenging task. Bayesian models offer a mathematical framework to reason
about model uncertainty. Variational methods have been used for approximating
intractable integrals that arise in Bayesian inference for neural networks. In
this report, we review the major variational inference concepts pertinent to
Bayesian neural networks and compare various approximation methods used in
literature. We also talk about the applications of variational bayes in
Reinforcement learning and continual learning.

negative:
Deep learning models are increasingly used in mobile applications as critical
components. Unlike the program bytecode whose vulnerabilities and threats have
been widely-discussed, whether and how the deep learning models deployed in the
applications can be compromised are not well-understood since neural networks
are usually viewed as a black box. In this paper, we introduce a highly
practical backdoor attack achieved with a set of reverse-engineering techniques
over compiled deep learning models. The core of the attack is a neural
conditional branch constructed with a trigger detector and several operators
and injected into the victim model as a malicious payload. The attack is
effective as the conditional logic can be flexibly customized by the attacker,
and scalable as it does not require any prior knowledge from the original
model. We evaluated the attack effectiveness using 5 state-of-the-art deep
learning models and real-world samples collected from 30 users. The results
demonstrated that the injected backdoor can be triggered with a success rate of
93.5%, while only brought less than 2ms latency overhead and no more than 1.4%
accuracy decrease. We further conducted an empirical study on real-world mobile
deep learning apps collected from Google Play. We found 54 apps that were
vulnerable to our attack, including popular and security-critical ones. The
results call for the awareness of deep learning application developers and
auditors to enhance the protection of deployed models.

negative:
We offer a general theoretical framework for brain and behavior that is
evolutionarily and computationally plausible. The brain in our abstract model
is a network of nodes and edges. Although it has some similarities to standard
neural network models, as we show, there are some significant differences. Both
nodes and edges in our network have weights and activation levels. They act as
probabilistic transducers that use a set of relatively simple rules to
determine how activation levels and weights are affected by input, generate
output, and affect each other. We show that these simple rules enable a
learning process that allows the network to represent increasingly complex
knowledge, and simultaneously to act as a computing device that facilitates
planning, decision-making, and the execution of behavior. By specifying the
innate (genetic) components of the network, we show how evolution could endow
the network with initial adaptive rules and goals that are then enriched
through learning. We demonstrate how the developing structure of the network
(which determines what the brain can do and how well) is critically affected by
the co-evolved coordination between the mechanisms affecting the distribution
of data input and those determining the learning parameters (used in the
programs run by nodes and edges). Finally, we consider how the model accounts
for various findings in the field of learning and decision making, how it can
address some challenging problems in mind and behavior, such as those related
to setting goals and self-control, and how it can help understand some
cognitive disorders.

negative:
The transfer learning toolkit wraps the codes of 17 transfer learning models
and provides integrated interfaces, allowing users to use those models by
calling a simple function. It is easy for primary researchers to use this
toolkit and to choose proper models for real-world applications. The toolkit is
written in Python and distributed under MIT open source license. In this paper,
the current state of this toolkit is described and the necessary environment
setting and usage are introduced.

negative:
This paper is a review of the evolutionary history of deep learning models.
It covers from the genesis of neural networks when associationism modeling of
the brain is studied, to the models that dominate the last decade of research
in deep learning like convolutional neural networks, deep belief networks, and
recurrent neural networks. In addition to a review of these models, this paper
primarily focuses on the precedents of the models above, examining how the
initial ideas are assembled to construct the early models and how these
preliminary models are developed into their current forms. Many of these
evolutionary paths last more than half a century and have a diversity of
directions. For example, CNN is built on prior knowledge of biological vision
system; DBN is evolved from a trade-off of modeling power and computation
complexity of graphical models and many nowadays models are neural counterparts
of ancient linear models. This paper reviews these evolutionary paths and
offers a concise thought flow of how these models are developed, and aims to
provide a thorough background for deep learning. More importantly, along with
the path, this paper summarizes the gist behind these milestones and proposes
many directions to guide the future research of deep learning.

negative:
Recent advances in artificial intelligence and the increasing need for
powerful defensive measures in the domain of network security, have led to the
adoption of deep learning approaches for use in network intrusion detection
systems. These methods have achieved superior performance against conventional
network attacks, which enable the deployment of practical security systems to
unique and dynamic sectors. Adversarial machine learning, unfortunately, has
recently shown that deep learning models are inherently vulnerable to
adversarial modifications on their input data. Because of this susceptibility,
the deep learning models deployed to power a network defense could in fact be
the weakest entry point for compromising a network system. In this paper, we
show that by modifying on average as little as 1.38 of the input features, an
adversary can generate malicious inputs which effectively fool a deep learning
based NIDS. Therefore, when designing such systems, it is crucial to consider
the performance from not only the conventional network security perspective but
also the adversarial machine learning domain.

negative:
What should I work on first? What can wait until later? Which projects should
I prioritize and which tasks are not worth my time? These are challenging
questions that many people face every day. People's intuitive strategy is to
prioritize their immediate experience over the long-term consequences. This
leads to procrastination and the neglect of important long-term projects in
favor of seemingly urgent tasks that are less important. Optimal gamification
strives to help people overcome these problems by incentivizing each task by a
number of points that communicates how valuable it is in the long-run.
Unfortunately, computing the optimal number of points with standard dynamic
programming methods quickly becomes intractable as the number of a person's
projects and the number of tasks required by each project increase. Here, we
introduce and evaluate a scalable method for identifying which tasks are most
important in the long run and incentivizing each task according to its
long-term value. Our method makes it possible to create to-do list gamification
apps that can handle the size and complexity of people's to-do lists in the
real world.

negative:
Probability theory, epistemically interpreted, provides an excellent, if not
the best available account of inductive reasoning. This is so because there are
general and definite rules for the change of subjective probabilities through
information or experience; induction and belief change are one and same topic,
after all. The most basic of these rules is simply to conditionalize with
respect to the information received; and there are similar and more general
rules. 1 Hence, a fundamental reason for the epistemological success of
probability theory is that there at all exists a well-behaved concept of
conditional probability. Still, people have, and have reasons for, various
concerns over probability theory. One of these is my starting point:
Intuitively, we have the notion of plain belief; we believe propositions2 to be
true (or to be false or neither). Probability theory, however, offers no formal
counterpart to this notion. Believing A is not the same as having probability 1
for A, because probability 1 is incorrigible3; but plain belief is clearly
corrigible. And believing A is not the same as giving A a probability larger
than some 1 - c, because believing A and believing B is usually taken to be
equivalent to believing A & B.4 Thus, it seems that the formal representation
of plain belief has to take a non-probabilistic route. Indeed, representing
plain belief seems easy enough: simply represent an epistemic state by the set
of all propositions believed true in it or, since I make the common assumption
that plain belief is deductively closed, by the conjunction of all propositions
believed true in it. But this does not yet provide a theory of induction, i.e.
an answer to the question how epistemic states so represented are changed
tbrough information or experience. There is a convincing partial answer: if the
new information is compatible with the old epistemic state, then the new
epistemic state is simply represented by the conjunction of the new information
and the old beliefs. This answer is partial because it does not cover the quite
common case where the new information is incompatible with the old beliefs. It
is, however, important to complete the answer and to cover this case, too;
otherwise, we would not represent plain belief as conigible. The crucial
problem is that there is no good completion. When epistemic states are
represented simply by the conjunction of all propositions believed true in it,
the answer cannot be completed; and though there is a lot of fruitful work, no
other representation of epistemic states has been proposed, as far as I know,
which provides a complete solution to this problem. In this paper, I want to
suggest such a solution. In [4], I have more fully argued that this is the only
solution, if certain plausible desiderata are to be satisfied. Here, in section
2, I will be content with formally defining and intuitively explaining my
proposal. I will compare my proposal with probability theory in section 3. It
will turn out that the theory I am proposing is structurally homomorphic to
probability theory in important respects and that it is thus equally easily
implementable, but moreover computationally simpler. Section 4 contains a very
brief comparison with various kinds of logics, in particular conditional logic,
with Shackle's functions of potential surprise and related theories, and with
the Dempster - Shafer theory of belief functions.

negative:
There is knowledge. There is belief. And there is tacit agreement.' 'We may
talk about objects. We may talk about attributes of the objects. Or we may talk
both about objects and their attributes.' This work inspects tacit agreements
on assumptions about the relation between objects and their attributes, and
studies a way of expressing them, presenting as the result what we term gradual
logic in which the sense of truth gradually shifts. It extends classical logic
instances with a new logical connective capturing the object-attribute
relation. A formal semantics is presented. Decidability is proved. Para-
consistent/epistemic/conditional/intensional/description/combined logics are
compared.

negative:
We argue that logical semantics might have faltered due to its failure in
distinguishing between two fundamentally very different types of concepts:
ontological concepts, that should be types in a strongly-typed ontology, and
logical concepts, that are predicates corresponding to properties of and
relations between objects of various ontological types. We will then show that
accounting for these differences amounts to the integration of lexical and
compositional semantics in one coherent framework, and to an embedding in our
logical semantics of a strongly-typed ontology that reflects our commonsense
view of the world and the way we talk about it in ordinary language. We will
show that in such a framework a number of challenges in natural language
semantics can be adequately and systematically treated.