text:
\section{Introduction}\label{sec:introduction}

Artificial neural networks achieve impressive results for various modeling tasks~\citep{lecun2015deep, wang2020recent}.
However, a downside of their superior performance and sophisticated structure is the comprehensibility of the learned models.
In many domains, it is crucial to understand the function learned by a neural network, especially when it comes to decisions that affect people~\citep{samek2019, molnar2020interpretable}.

A common approach to tackle the problem of  interpretability without sacrificing the superior performance is using a surrogate model as gateway to interpretability \citep{molnar2020interpretable}.
Most existing global surrogate approaches use a distillation procedure to learn the surrogate model based on the predictions of the neural network~\citep{molnar2020interpretable, frosst2017distilling}. Therefore, they query the neural network based on a representative set of samples and the resulting input-output pairs are then used to train the surrogate model. This representative sample usually comprises the training data of the original model, or at least follows its distribution~\citep{molnar2020interpretable, lopes2017data}. However, there are many cases where the training data cannot easily be exposed due to privacy or safety concerns~\citep{lopes2017data,bhardwaj2019dream,nayak2019zero}.
Without having access to the training data, traditional approaches can fail to provide meaningful explanations since the querying strategy can easily miss dense regions of the training data such that the resulting samples are a poor approximation of the true function, as we will show in the following example. \\%since the relevant parts are not properly queried.

\emph{Example 1.}
The Credit Card Default dataset~\citep{dataset_credit_card} comprises personal, confidential data which usually cannot be exposed to external authorities.
The task is to predict whether a client will default the payment in the current month, which can be solved efficiently using neural networks.
To gain insight into the decision-making process of the neural network, we can learn a global surrogate model.
Unfortunately, if the training data is not accessible, we can't ensure that the neural network is properly queried, and the explanation contains the relevant information when using a traditional, sample-based distillation.

Figure~\ref{fig:vanilla_credit_card_reduced_sample} shows such a scenario, where the explanation generated by a sample-based distillation without training data contains a misconception: It encodes the rule that we should always predict \emph{No Default} if the payment amount of the last month is larger than $373,000$ without taking the payment history of the client into account.
This mismatch between network and surrogate model is also reflected in the low fidelity between the network and surrogate model on the training data. However, in reality, this fidelity cannot be computed when the training data is not available, such that these misconceptions might go unnoticed. This can lead to wrong assumptions about what the network actually learned. \\

\begin{figure}[htb]
\centering
\begin{subfigure}{.5\columnwidth}
   \centering
  \includegraphics[width=\columnwidth]{./DT_example_sample_reduced.pdf}
  \caption{Sample-Based Decision Tree}
  \label{fig:vanilla_credit_card_reduced_sample}

\end{subfigure}%
\begin{subfigure}{.5\columnwidth}
  \centering
  \includegraphics[width=\columnwidth]{./DT_example_inet_reduced.pdf}
  \caption{\mbox{$\mathcal{I}$-Net} Decision Tree}
  \label{fig:vanilla_credit_card_reduced_inet}
\end{subfigure}
\caption{\textbf{Explaining Neural Networks for Credit Card Default Prediction.} The DT on the left is learned by a sample-based distillation without access to training data, and the DT on the right is predicted by the \mbox{$\mathcal{I}$-Net}. The \mbox{$\mathcal{I}$-Net} makes reasonable splits and achieves a significantly higher fidelity on the real data.}
\label{fig:vanilla_credit_card_reduced}
\end{figure}

As shown in \emph{Example 1}, knowing the training data is crucial for sample-based methods and without them, it is often not possible to generate reasonable explanations.
Recent approaches tackle this issue by using only a subset of the training data and/or layer activations to generate a representative set of samples~\citep{lopes2017data,bhardwaj2019dream,nayak2019zero}. However, they still rely on a proper querying of the model and use a sample-based distillation.

In contrast, the \mbox{$\mathcal{I}$-Net} approach introduced by \citet{marton2022explanations} is a sample-free approach that only accesses the network parameters and therefore does not rely on a proper querying.
This is achieved by using a second neural network (the so-called \mbox{$\mathcal{I}$-Net}) which learns a mapping from the network parameters to a human-understandable representation of the network function.
Following this approach, we can generate reasonable explanations, even when the training data is not accessible, as shown in Figure~\ref{fig:vanilla_credit_card_reduced_inet}: The \mbox{$\mathcal{I}$-Net} achieves a high fidelity and the resulting surrogate model encodes the rule that \emph{Default} is predicted if the payment for the last month was delayed for more than two months, which is a reasonable explanation for this scenario.

The \mbox{$\mathcal{I}$-Net} was originally devised for lower-order polynomials as a surrogate model. While polynomials can be reasonable explanations for regression tasks, they are not well-suited for representing decision boundaries of a classification task, which is the focus of our paper. In contrast, decision trees (DTs) are frequently used as an explainable model for classification tasks since they make hierarchical decisions and therefore are easy to comprehend for humans~\citep{frosst2017distilling}.
In the recent literature, soft DTs (SDTs) are successfully used as interpretable surrogate models~\citep{frosst2017distilling}. While SDTs make multivariate splits, they usually achieve a higher fidelity than standard DTs, but also have a higher level of complexity.

In this paper, we make the following contributions:
\begin{itemize}
    \item We propose an improved data generation method for the training of an \mbox{$\mathcal{I}$-Net} (Section~\ref{sssec:data_generation}) which allows more robust and distribution-independent explanations on real-world datasets.
    \item We present an \mbox{$\mathcal{I}$-Net} design that is able to represent standard DTs and SDTs as surrogate model (Section~\ref{ssec:inets_dts}) with a high fidelity.

    \item We introduce univariate SDTs as complexity-fidelity trade-off (Section~\ref{sssec:inets_dts_soft}) and show that they can be implemented efficiently using the \mbox{$\mathcal{I}$-Net}.
    \item We empirically evaluate our approach against sample-based approaches for learning standard DTs and SDTs and show that it achieves superior results when training data is not accessible (Section~\ref{ssec:exp_results}).
\end{itemize}

\section{\mbox{$\mathcal{I}$-Nets} as a Sample-Free Approach to Global Model Interpretability}\label{sec:inets}
In this section, we summarize the task of explaining neural networks, focusing on the case where the networks' training data is not available, followed by a brief introduction to the \mbox{$\mathcal{I}$-Net} approach. For a more in-depth explanation of the \mbox{$\mathcal{I}$-Net} approach, we refer to \citet{marton2022explanations}.

\subsection{Global Explanations for Neural Networks}\label{ssec:explaining_nn}

The general task of globally explaining neural networks can be formalized as finding a function $g: X \rightarrow P(Y \lvert X)$ (i.e., a surrogate model) that approximates the decision function of a neural network  $\lambda: X \rightarrow P(Y \lvert X)$, such that $\forall \mathbf{x} \in X: \lambda(\mathbf{x}) \approx g(\mathbf{x})$, where $X$ is a set of feature vectors and $Y$ is a set of classes.

Since the \mbox{$\mathcal{I}$-Net} approach implements a learning task, it is convenient to distinguish between the functions $\lambda$ and $g$ and their representations $\theta_\lambda \in \Theta_\lambda$ and $\theta_g \in \Theta_g$~\citep{marton2022explanations}. The representation $\theta_\lambda$ consists of the network parameters, i.e., the weights and biases of the neural network. Similarly, $\theta_g$ is the parameter vector of the surrogate model and depends on the selected function family.

The process of generating explanations can be formalized as a function $\mathcal{I}: \Theta_\lambda \rightarrow \Theta_g$ that maps representations of $\lambda$ to representations of $g$~\citep{marton2022explanations}.
Traditional approaches for generating global surrogate models post-hoc implement $\mathcal{I}$ via a sample-based procedure, as shown in Figure~\ref{fig:decision-boundary-explanation} (\RNum{1}-\RNum{2}).
They generate a new dataset, where the labels are obtained by querying $\lambda$ based on a set of data points. In the next step, a surrogate model is trained using the generated dataset, maximizing the fidelity between $\lambda$ and $g$.
As shown by \citet{marton2022explanations}, this process is time-consuming, which can be a huge drawback if timely explanations are required, as for instance in an online learning scenario.
Additionally, it also strongly depends on the data used for querying the model.
Information that is not properly queried cannot be contained in the explanation, as already shown in \emph{Example 1}. Therefore, in the literature it is suggested to use the original train data or data from the same distribution for querying the model~\citep{molnar2020interpretable, lopes2017data}, which usually yields to meaningful explanations as depicted in \RNum{1} in Figure~\ref{fig:decision-boundary-explanation}. However, if the training data is not accessible or not existing anymore, the model has to be queried based on some sampled data (\RNum{2} in Figure~\ref{fig:decision-boundary-explanation}).
In this case, it is often not possible to generate meaningful explanations with sample-based approaches, since we cannot ensure a proper querying and therefore the explanation does not necessarily focus on the relevant aspects.

\begin{figure}[tb]

    \includegraphics[width=0.99\columnwidth]{./decision_boundary_explanation.pdf}
    \caption{\textbf{Sample-Based and $\boldsymbol{\mathcal{I}}$-Net Approach.} Sample-based approaches query the target network based on a set of data points. Using the train data to query the network (\RNum{1}) usually generates a meaningful explanation. If the training data is not available, we have to query the network based on randomly sampled data, e.g., from a uniform distribution (\RNum{2}), which often cannot generate a meaningful explanation since relevant parts are not queried properly. The \mbox{$\mathcal{I}$-Net} uses the network parameters as an input to generate a reasonable explanation (\RNum{3}) and does not rely on querying the neural network.}

    \label{fig:decision-boundary-explanation}
\end{figure}

\subsection{Reasonable Explanations}\label{ssec:reasonable_explanations}
In the following, we discuss what constitutes a meaningful explanation for a neural network.
In general, the decision boundary of the surrogate model should closely match the decision boundary of the network we want to interpret to achieve a high fidelity. However, we argue that it is necessary to also take the data distribution into account: A decision boundary should assign as many samples as possible to the correct class. Therefore, it is crucial that the decision boundary is composed correctly in the areas where many samples are located.
Accordingly, for a reasonable explanation, the decision boundary should match the model we want to interpret especially in regions where many samples are located, while it is less important that the decision boundaries match in regions with low data density.
In other words, we are less interested in an explanation that shows us how the model behaves when making predictions on data points that do not occur in reality.
This concept is visualized in Figure~\ref{fig:decision-boundary-explanation_simple}.
\begin{figure}[tb]
    \centering
    \includegraphics[width=0.80\columnwidth]{./decision_boundary_explanation_simple.pdf}
    \caption{\textbf{Good and Bad Explanations.} This figure shows an exemplary decision boundary of a bad (\RNum{2}) and a good (\RNum{3}) explanation for the model we want to interpret (\RNum{1}). Without considering the data (a), the explanation shown in \RNum{2} appears very reasonable, since the areas created by the decision boundary cover most of the decision boundary of the original model. However, when taking the data into account (b), we can see that the small area in the center of the picture is very important, since there are many samples located. This is neglected by the explanation shown in \RNum{2} and only considered by the explanation shown in \RNum{3}.}

    \label{fig:decision-boundary-explanation_simple}
\end{figure}
In Section~\ref{sssec:visual_eval}, we show that traditional, sample-based approaches cannot generate such reasonable explanations when the training data is not available.

\subsection{Explanations for Neural Networks by Neural Networks}
To renounce the dependency on a proper querying of the model, we can implement $\mathcal{I}$ as a neural network as proposed by \citet{marton2022explanations}. Therefore, we transform the task of explaining neural networks into a machine learning task, as shown in Figure~\ref{fig:decision-boundary-explanation}~\RNum{3}.
The concept of \mbox{$\mathcal{I}$-Nets} is depicted more detailed in Figure~\ref{fig:overview} and involves two major steps:
\begin{enumerate}
    \item We train a set of neural networks on synthetic data and extract their learned parameters.
    \item We train a second neural network, the \mbox{$\mathcal{I}$-Net}, using the parameters extracted in the first step as input data.
\end{enumerate}

Thereby, no supervision in terms of actual labels is required during the training. Instead, the fidelity between $\lambda$ and $g$ is computed using a distance measure over a set of data points in the loss function. Since the loss is only computed during the training, no data except the network parameters is required when applying the \mbox{$\mathcal{I}$-Net}. This is a major advantage to sample-based approaches, where the training data is required for each network we want to interpret.
Accordingly, to generate an explanation, \mbox{$\mathcal{I}$-Nets} only need access to the network parameters and therefore, the approach can be applied in scenarios where the training data is not accessible without suffering a performance deficit.

\begin{figure}[tb]

    \includegraphics[width=0.99\columnwidth]{./simple_approach.pdf}
    \caption{\textbf{Overview of the \mbox{$\boldsymbol{\mathcal{I}}$-Net}  Approach.} The neural network parameters as input are translated into a surrogate model~\cite{marton2022explanations}.}

    \label{fig:overview}
\end{figure}

The most crucial part of the \mbox{$\mathcal{I}$-Net} approach is an efficient training procedure. Thereby, as for most machine learning tasks, good training data (in our case, a set of network parameters $\Theta_\lambda$) is important.
Therefore, we present an improved data generation method making \mbox{$\mathcal{I}$-Nets} applicable for real world scenarios in Section~\ref{sssec:data_generation}.

\section{Robust \mbox{$\mathcal{I}$-Nets} for Decision Trees}\label{ssec:inets_improvements}

In this section, we present the main contributions of this paper.
\citet{marton2022explanations} argue that \mbox{$\mathcal{I}$-Nets} can be trained solely based on synthetic data. However, it is crucial that this synthetic data comprises reasonable learning problems to assure that an application of the \mbox{$\mathcal{I}$-Net} is possible in a real-world setting. Therefore, we will introduce an improved data generation method that considers multiple data distributions and creates reasonable learning tasks (Section~\ref{sssec:data_generation}).
Furthermore, \citet{marton2022explanations} focus on regression tasks. In contrast, we focus on classification tasks and therefore present an adjusted loss function for the \mbox{$\mathcal{I}$-Net} (Section~\ref{sssec:loss}).

In general, the \mbox{$\mathcal{I}$-Net} framework can be applied to arbitrary function families for $g$, as long as a suitable representation $\theta_g$ is available.
In Section~\ref{ssec:inets_dts}, we introduce different DT variants and propose corresponding representations $\theta_g$ that allow an efficient training.

\subsection{Improved Data Generation and Training Procedure}\label{sssec:improvements}

\subsubsection{Data Generation Method}\label{sssec:data_generation}
The data generation method proposed by \citet{marton2022explanations} focused on maximizing the performance of the \mbox{$\mathcal{I}$-Net} during training by learning functions $\lambda$ that are similar to the function family of $g$. This is achieved by randomly sampling a set of functions from the family of $g$. These functions are queried to generate labels for a uniformly sampled dataset, which is used to learn $\lambda$.
This procedure ensures that the functions $\lambda$ are representative of $g$, which enables efficient training.
However, a high training performance does not necessarily mean that the model generalizes well to unseen data, i.e., neural networks trained on real-world datasets.

Additionally, \citet{marton2022explanations} use a uniform data distribution to query $\lambda$ for the fidelity calculation in the \mbox{$\mathcal{I}$-Net} loss.
However, if we only consider a uniform distribution during the training of the \mbox{$\mathcal{I}$-Net}, we might not be able to make reasonable predictions if the network we want to interpret was trained using data from a substantially different distribution, as we will show in Section~\ref{sssec:improvements_eval}.
This problem is related to the general problem that occurs for a machine learning task, if the data we are actually interested in (i.e., the test data) is from a different distribution than the data used for training the model.

To tackle this issue, we propose using multiple, different distributions during the training of the \mbox{$\mathcal{I}$-Net} to make it more robust and therefore applicable on real-world datasets. In this process, we can also utilize the fact that an \mbox{$\mathcal{I}$-Net} can be trained in a controlled, synthetic environment~\citep{marton2022explanations}: For each $\theta_\lambda \in \Theta_\lambda$, we know the data that was used for learning $\lambda$. Therefore, we can use these data points to compute the \mbox{$\mathcal{I}$-Net} loss on a meaningful set of samples during the training. The \mbox{$\mathcal{I}$-Net} utilizes this additional knowledge to generalize. Since the loss is only calculated during training, it can generate meaningful explanations solely based on the network parameters $\theta_\lambda$ at test time.

In general, generating the training data for the \mbox{$\mathcal{I}$-Net} $\Theta_\lambda$ involves three major steps:
\begin{enumerate}
    \item Generate $N$ datasets $\mathcal{D}_\lambda = \{(\mathbf{x}^{(j)},y^{(j)})\}_{j=1}^M$, each comprising $M$ samples.
    \item For each dataset $\mathcal{D}_\lambda$, train a network $\lambda$, extract the network parameters $\theta_\lambda$ and add them to the training set $\Theta_\lambda$.
    \item Use $\Theta_\lambda$ to train an \mbox{$\mathcal{I}$-Net} for the respective function family.
\end{enumerate}

\begin{table}[tb]
\centering
\begin{tabular}{cccc}
\toprule
\textbf{Distribution} & \textbf{Parameter} $\boldsymbol{p_1}$ & \textbf{Parameter} $\boldsymbol{p_2}$ & \textbf{Symbol} \\ \midrule
Uniform & minimum & maximum & $\mathcal{U}(p_1, p_2)$ \\
Normal & location (mean) & scale (standard deviation) & $\mathcal{N}(p_1, p_2)$ \\
Gamma & shape ($k$) & scale ($\theta$) & $\Gamma(p_1, p_2)$  \\
Beta & $\alpha$ & $\beta$ & $\text{B}(p_1, p_2)$  \\
Poisson & lambda & - & $\text{Poi}(p_1)$ \\ \bottomrule
\end{tabular}
\caption{\textbf{Data Generation Distributions.} This table summarizes the different distributions used for the data generation of the \mbox{$\mathcal{I}$-Net} with their parameters and used symbol.}
\label{tab:distrib}
\end{table}

\begin{figure}[tb]

    \includegraphics[width=0.99\columnwidth]{./distributions.pdf}
    \caption{\textbf{Data Generation Visualization.} This graphic visualizes the generation of a balanced, random dataset used for training a network $\lambda$ where $D \in \{\mathcal{U}, \mathcal{N}, \Gamma, \text{B}, \text{Poi}\}$. For each feature, a random distribution with two random parametrizations is chosen and a random number of data points is sampled from each distribution.}
    \label{fig:distrib}
\end{figure}

The data generation is visualized in Figure~\ref{fig:distrib}:
For each feature $i$, we sample data points from one distribution with $k$ different parametrizations, where $k$ is the number of classes. For this paper, we focus on binary classification tasks and therefore set $k=2$. The distribution $D_{i,k}$ is sampled uniformly from $\{\mathcal{U}, \mathcal{N}, \Gamma, \text{B}, \text{Poi}\}$ for each feature. The distributions considered within this paper are summarized in Table~\ref{tab:distrib}. The distributions were selected to cover a wide range of diverse distributions that are reasonable for many different real-world phenomena~\citep{distribution_relation, mun2015understanding}.
The parametrization for the distributions $D_{i,0}$ and $D_{i,1}$ are again randomly drawn from $\mathcal{U}(0,p)$, where $p$ is a hyperparameter for the data generation procedure.
The number of samples is selected randomly, where $M_0 =  \lceil \mathcal{U}(1,M-1) \rceil$ data points are sampled from $D_{i,0}$ and  $M_1 = M - M_0$ data points are sampled from $D_{i,1}$. The generated datasets are balanced and for each feature and the first $\frac{M}{2}$ data points are associated with \emph{Class 0} and the subsequent $\frac{M}{2}$ data points are associated with \emph{Class 1}. This procedure is formalized in Algorithm~\ref{alg:generate-data}.
We can see the proposed data generation method as a generalization of common, synthetic machine learning problems (as for instance make\_blobs\footnote{Accessible using sklearn under \url{https://scikit-learn.org/0.15/modules/generated/sklearn.datasets.make_blobs.html} (Accessed 15.05.2022)}), that is able to generate more realistic tasks.

\begin{algorithm}[tb]
\caption{Generate Multi-Distribution Data}   \label{alg:generate-data}

\begin{algorithmic}[1]
    \Function{generate}{$n,D,M$}
            \For{$i = 1,\dots,n$}
                \State{$D_i \sim \mathcal{U}\{\mathcal{U}, \mathcal{N}, \Gamma, \text{B}, \text{Poi}\}$} \Comment{Randomly chose distribution}
                \State{$M_0 \sim \lceil \mathcal{U}(1,M-1) \rceil$}

                \State{$\mathbf{p_0} \sim \mathcal{U}(0,p)$} \Comment{Randomly sample parameters for $D_i,0$}
                \State{$\mathbf{p_1} \sim \mathcal{U}(0,p)$} \Comment{Randomly sample parameters for $D_i,1$}
                \For{$j = 1,\dots,M_0$}
                    \State{$x_i^{(k)} \sim D_i(\mathbf{p_0})$}
                \EndFor
                \For{$j = 1,\dots,M-M_0$}
                    \State{$x_{i}^{(M_0+j)} \sim D_i(\mathbf{p_1})$}
                \EndFor

                \State{$\mathbf{x}_i \leftarrow \frac{\mathbf{x}_i - min(\mathbf{x}_i)}{max(\mathbf{x}_i) - min(\mathbf{x}_i})$}
                \Comment{Scale feature to $[0,1]$}
   	        \EndFor

            \For{$j = 1,\dots, \lceil \frac{M}{2} \rceil$}
   	            \State{$y^{(j)} \leftarrow 0$}
            \EndFor
            \For{$j = 1,\dots, \lfloor \frac{M}{2} \rfloor$}
   	            \State{$y^{(j)} \leftarrow 1$}
            \EndFor
            \State $\mathcal{D} \leftarrow \{\mathbf{x}^{(j)},y^{(j)}\}_{j=1}^{M}$
            \State \Return{$\mathcal{D}$}
    \EndFunction
\end{algorithmic}

\end{algorithm}

\subsubsection{Adjusted Loss Function}\label{sssec:loss}
While \citet{marton2022explanations} focused on regression tasks, we focus on binary classification tasks within this paper.
Therefore, the loss function has to be adjusted by using binary cross-entropy as distance measure to quantify the fidelity between $\lambda$ and $g$:

\begin{equation}\label{eq:bc_inet}
\begin{split}

    \text{BC}(\theta_\lambda,\theta_g) = \frac{1}{M} \sum_{j=1}^M & \lfloor \lambda(\mathbf{x}^{(j)}) \rceil \times  log(g(\mathbf{x}^{(j)}) ) \\
     + & (1 - \lfloor \lambda(\mathbf{x}^{(j)}) \rceil) \times log(1 - g(\mathbf{x}^{(j)}) )
\end{split}
\end{equation}

Here, $\lfloor \cdot \rceil$ denotes rounding to the closest integer, which is required to calculate the binary cross-entropy correctly.
The \mbox{$\mathcal{I}$-Net} loss for a set of network parameters $\Theta_\lambda = \{\theta_\lambda^{(i)}\}_{i=1}^N$ is then computed as

\begin{equation}\label{eq:loss_inet}
    \mathcal{L}_{\mathcal{I}} = \frac{1}{\lvert \Theta_\lambda \rvert} \sum_{\theta_\lambda \in \Theta_\lambda} \text{BC}(\theta_\lambda,\mathcal{I}(\theta_\lambda)).
\end{equation}

\subsection{Function Families and \mbox{$\mathcal{I}$-Net} Output Representation}\label{ssec:inets_dts}

\subsubsection{\mbox{$\mathcal{I}$-Nets} for Standard Decision Trees}\label{sssec:inets_dts_standard}
The first function family we will consider as surrogate models are standard DTs. DTs and decision rules are frequently used as explanations, since they are comparatively easy to understand for most humans~\citep{molnar2020interpretable}.

\paragraph{Standard Decision Tree Representation in the \mbox{$\mathcal{I}$-Net} Framework}
The \mbox{$\mathcal{I}$-Net} approach requires a suitable representation $\theta_g$ for standard DTs to enable efficient learning. Specifically, we need a one-dimensional encoding of internal and leaf nodes, as shown in Figure~\ref{fig:inet_dt}.

\begin{figure}[tb]
    \centering
    \includegraphics[width=0.8\columnwidth]{./inet_output_dt.pdf}
    \caption{\textbf{Exemplary $\boldsymbol{\mathcal{I}}$-Net Output for DTs.} The DT representation is predicted by the \mbox{$\mathcal{I}$-Net} using three separate output layers with different activation functions. The output shows an exemplary DT of depth two for a binary classification task on a dataset with two features.}
    \label{fig:inet_dt}
\end{figure}

The inner node of a DT comprises two major parts: The first part is the feature that is considered within the split, and the second part is the split value. The operator is fixed to less ($<$) as it is common practice for representing DTs. Furthermore, we can fix the left path to be the true path and the right path as the false path.

The feature $x_i$ considered in the split can be defined by enumerating the features, where $i \in \{0,1, \dots ,n\}$. We can represent this using $n$ neurons and a softmax activation for each inner node (i.e., we can see it as a classification task for which feature to consider at a certain split).

For the split value, we can assume that all features are scaled to be within $[0, 1]$, as it is common practice. To represent this in the \mbox{$\mathcal{I}$-Net} output, we can use sigmoid activations, to constraint the output interval. However, due to the functional form of the sigmoid activation, the \mbox{$\mathcal{I}$-Net} prefers split values close to $0.5$. To counteract this tendency, we used a squeezed sigmoid activation, which we define as $\frac{1}{1+e^{-3x}}$. This supports the \mbox{$\mathcal{I}$-Net} in choosing more distinct split values.
Furthermore, the output layer does not comprise one split value for each split, but $n$ split values for each split (one for each feature). To construct the DT, we always use the split value at the index indicated by the feature identifier. This design choice is influenced by the fact that we always need to consider the meaning of a split value in context with the corresponding feature. In other words, while the split value $0.7$ might be a reasonable threshold for the feature $x_0$, it might not be reasonable at all for the feature $x_1$. Designing the \mbox{$\mathcal{I}$-Net} output with one split value for each feature and each inner node, we can make this interaction easier to learn.

In a standard DT, the leaf nodes comprise the decision for a certain path (i.e., the class to be predicted). However, to compute the \mbox{$\mathcal{I}$-Net} loss in Equation~\ref{eq:loss_inet}, it is necessary that $g$ has probabilities as an output. Therefore, we adjust the DTs to not just have a class in the leaf node, but a probability. This is similar to the purity in the leaf node of a DT, which is also often used as a gateway to predicting probabilities using a standard DT. In a binary classification case, we can use a single value to represent the probabilities of predicting \emph{Class 1} and thereby, the probability of \emph{Class 0} is the complementary probability. In the output layer of the \mbox{$\mathcal{I}$-Net}, we can represent this using a total of $2^d$ neurons with sigmoid activations (one neuron for each leaf node).
This can easily be extended for a multi-class classification problem with $k$ classes by using $k \times 2^d$ neurons and one softmax activation over $k$ neurons (one softmax for each leaf node).

\subsubsection{\mbox{$\mathcal{I}$-Nets} for Soft Decision Trees}\label{sssec:inets_dts_soft}

SDTs were proposed by \citet{frosst2017distilling} to overcome the interpretability problem that arises from distributed hierarchical representations when using neural networks by expressing the knowledge using hierarchical decisions of a tree-based structure.
Unlike standard DTs, SDTs do not make hard true/false splits at each internal node, but use soft decisions associated with probabilities for each path. In the following, we will shortly introduce the functioning of SDTs. For a more in-depth description, especially concerning the learning algorithm, we refer to \citet{frosst2017distilling}.

Figure~\ref{fig:sdt} shows a SDT with a single internal node following the design of \citet{frosst2017distilling}. Each internal node $j$ comprises a filter $\mathbf{w}^j$ and a bias $b^j$. While the bias is a single, learned value, the filter consists of one value for each feature. Accordingly, in contrast to a standard DT with univariate decisions, a SDT has a multivariate decision at each internal node. This comes with a significantly higher model complexity, especially with an increasing number of features.

At each internal node, the probability of taking the right branch is calculated by
\begin{equation}
    P^j(\mathbf{x}) = S(\mathbf{x}w^j + b^j)
\end{equation}
where $x$ is an input sample and $S$ is a sigmoid function defined as $S(x) = \frac{1}{1+e^{-x}}$.
Each leaf nodes $l$ comprise a probability distribution $Q^l_.$, which for the binary case is defined as
\begin{equation}
    Q^l_k = \frac{e^{\phi^l_k}}{e^{\phi^l_0} + e^{\phi^l_1}}
\end{equation}
where $k \in \{0,1\}$ is the output class and $\phi^l_.$ is a learned parameter for each leaf $l$.

Usually when using SDTs, there is not only a single leaf node considered when making a prediction, but all leaf nodes are multiplied with their path probabilities to calculate the final probability distribution. However, \citet{frosst2017distilling} suggest increasing the interpretability of SDTs by just considering the path with the maximum path probability when calculating the final probability distribution. Since this does not significantly affect the performance, we will only consider SDTs using the maximum path probability in this paper.

\begin{figure}[tb]
    \centering
    \includegraphics[width=0.70\columnwidth]{./SDT_reduced_hinton.pdf}
    \caption{\textbf{Soft Decision Tree} This figure shows a minimal SDT with a single inner node and two leaf nodes.}
    \label{fig:sdt}
\end{figure}

While \citet{frosst2017distilling} focused on image datasets for a better visualization of the explanation, they also reported that their algorithm can efficiently be applied to tabular data. However, for tabular data, especially when the feature space is high-dimensional, using SDTs as a surrogate model can make the explanation hard to understand for humans due to their high complexity. To counteract the strong correlation between the model complexity and the number of features, we propose univariate SDTs. Univariate SDTs are similar to standard SDTs, but the filter at each internal node comprises only one value $\neq 0$.
While the complexity of univariate SDTs is similar to standard DTs, they can represent decision boundaries that are not parallel to the feature axis. Simultaneously, they maintain the hierarchical structure, making them easier to comprehend for humans.

Unfortunately, univariate SDTs cannot be trained using the algorithm proposed by \citet{frosst2017distilling} without adjustments and there is to the best of our knowledge no existing learning algorithm for univariate SDTs.
However, we can constrain the learning algorithm to just consider the feature with the highest absolute value (which is also the value that according to the design has the highest influence on the decision) when calculating the path probabilities. We can implement this by multiplying the filter with a binary mask generated by an argmax or exaggerated softmax of the absolute filter values. Accordingly, during the forward pass, the filter values are calculated as:
\begin{equation}
    \mathbf{w}_i = \mathbf{w}_i \times \sigma(\beta_2 \times \lvert \mathbf{w}_i \rvert)
\end{equation}
By increasing the temperature $\beta_2$, we increase the extent to which the filter with the highest absolute value influences the calculation of the path probability. When $\beta_2$ is sufficiently high, we approximate an argmax and the path probability is calculated only based on the filter with the highest absolute value.

\paragraph{Soft Decision Tree Representations in the \mbox{$\mathcal{I}$-Net} Framework}

To use SDTs as surrogate models within the \mbox{$\mathcal{I}$-Net} framework, we again need a suitable representation $\theta_g$. Fortunately, the encoding for standard SDT shown in Figure~\ref{fig:inet_sdt} is straightforward: We can represent the internal nodes with $n$ output neurons for the filter (one for each feature) and one output neuron for the bias. Since there are no specific ranges for the filter and bias value in the SDT, we use linear activations. The same accounts for $\phi^l_.$, where we need $k$ output neurons for each leaf node. Again, we can use linear activations here, since the final probabilities are calculated by $Q^l_k$ and no specific range for $\phi^l_.$ is required.

\begin{figure}[tb]

    \includegraphics[width=0.95\columnwidth]{./inet_output_sdt.pdf}
    \caption{\textbf{Exemplary $\boldsymbol{\mathcal{I}}$-Net Output for SDTs.} The SDT representation is predicted by the \mbox{$\mathcal{I}$-Net} using three separate output layers with linear activation functions. The output shows an exemplary SDT of depth two for a binary classification task on a dataset with two features.}
    \label{fig:inet_sdt}
\end{figure}

Unfortunately, the representation of univariate SDTs as \mbox{$\mathcal{I}$-Net} output is not as straightforward. This is mainly caused by the fact that neural networks hardly predict a value of $0$, which would be necessary when using the same representation we proposed for standard SDTs. This would result in many small filter values and therefore still multivariate trees.
One solution is using a feature identifier and a separate filter value output, similar to the representation of the internal nodes for standard DTs (Figure~\ref{fig:inet_dt}).
Accordingly, we can use $n$ output neurons and a softmax activation for each internal node to indicate at which position to set the filter value (i.e., we can see this as a classification task for which filter value is $\neq 0$). We represent the filter value again using $n$ neurons with a linear activation for each internal node.
The bias and leaf parameters are adopted from the standard SDT. An exemplary representation for univariate SDTs is shown in Figure~\ref{fig:inet_sdt1}.
\begin{figure}[tb]

    \includegraphics[width=0.95\columnwidth]{./inet_output_sdt1.pdf}
    \caption{\textbf{Exemplary $\boldsymbol{\mathcal{I}}$-Net Output for Univariate SDTs.} The univariate SDT representation is predicted by the \mbox{$\mathcal{I}$-Net} using four separate output layers with different activation functions. The output shows an exemplary univariate SDT of depth two for a binary classification task on a dataset with two features.}
    \label{fig:inet_sdt1}
\end{figure}

\section{Evaluation} \label{sec:eval}
The goal of our evaluation is to show that \mbox{$\mathcal{I}$-Nets} are able to interpret neural networks trained on real-world datasets without requiring access to the training data, and achieve a higher fidelity than sample-based approaches in most of the cases. Therefore, we will address the following in our evaluation:
\begin{itemize}
    \item We illustrate which effects occur once the training data is not accessible for querying the model and thereby show that it is crucial for sample-based approaches to access the training data (Section~\ref{sssec:visual_eval}).
    \item We empirically investigate the fidelity of the \mbox{$\mathcal{I}$-Net} in comparison to sample-based approaches on real-world datasets if the training data is not accessible (Section~\ref{sssec:real_world_eval}).
    \item We perform an ablation study showing the effect of the improved data generation method introduced in Section~\ref{sssec:data_generation} on the \mbox{$\mathcal{I}$-Net} performance for real-world datasets (Section~\ref{sssec:improvements_eval}).
    \item We present a case study of credit card default prediction, comparing the explanations for a neural network generated by the \mbox{$\mathcal{I}$-Net} and sample-based approaches without access to the training data (Section~\ref{sssec:case_study_eval}).
\end{itemize}

\subsection{Experimental Setup}\label{ssec:experimental_setup}
Within our experiments, we compare \mbox{$\mathcal{I}$-Nets} with standard distillation approaches for a scenario where the original training data is not available.
Thereby, we used the representations $\Theta_g$ described in Section~\ref{ssec:inets_dts} for the \mbox{$\mathcal{I}$-Net}. The sample-based distillation was conducted as follows:

\begin{itemize}
    \item \textbf{Standard Decision Trees: } For standard DTs, we used the implementation from sklearn\footnote{Available under: \url{https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html}} (Accessed 15.05.2022) which uses the CART algorithm for DT induction~\citep{breiman1984classification}.
    \item \textbf{Univariate Soft Decision Trees: } For univariate SDTs, we used the learning algorithm introduced by \citet{frosst2017distilling} with our adjustments described in Section~\ref{sssec:inets_dts_soft}.
    \item \textbf{Standard Soft Decision Trees: } For SDTs, we used the algorithm proposed by \citet{frosst2017distilling}.
\end{itemize}
The hyperparameters used during our experiments are summarized in Table~\ref{tab:dt-params}-\ref{tab:sdt-params}.

Since we assume that the training data is not available, we needed to generate data for querying the neural network to distill a surrogate model. Therefore, we selected three sampling strategies for generating the query data as benchmarks:
\begin{enumerate}
    \item \textbf{Multi-Distribution: } According to Algorithm~\ref{alg:generate-data}, i.e., considering different data distributions to allow for a fair comparison with the \mbox{$\mathcal{I}$-Net}.
    \item \textbf{Standard Uniform: }  A standard uniform distribution $U(0,1)$.
    \item  \textbf{Standard Normal: } A standard normal distribution $\mathcal{N}(0, 1)$.
\end{enumerate}
For each sampling strategy, we sampled $10000$ data points. However, this is only necessary for the sample-based approaches and the \mbox{$\mathcal{I}$-Net} as sample-free approach does not rely on querying to generate explanations.
Increasing the number of sampling points further did not enhance the fidelity of sample-based approaches (see Figure~\ref{fig:sample_size}), but only increases their runtime.
We further assumed, that the data used for training the neural network was scaled to be in $[0, 1]$ and therefore, we also scaled the sampled data to the same interval.

The network parameters $\Theta_\lambda$ for training the \mbox{$\mathcal{I}$-Net} were generated according to Algorithm~\ref{alg:generate-data} using the hyperparameters in Table~\ref{tab:lambda-net-params}. The hyperparameter $p$ which defines the maximum value for the distribution parameters was fixed to $5$ during the data generation for all experiments. Furthermore, we excluded all datasets that were linearly separable during the data generation to focus on more complex and reasonable datasets.
The \mbox{$\mathcal{I}$-Net} hyperparameters are summarized in Table~\ref{tab:i-net-params} and were tuned using a greedy neural architecture search according to \citet{jin2019auto} followed by a manual fine-tuning of the selected values. We selected one \mbox{$\mathcal{I}$-Net} architecture for each of the three function families.

The neural networks considered during the evaluation in Section~\ref{ssec:exp_results} were trained with the hyperparameters summarized in Table~\ref{tab:lambda-net-params}. The fidelity between the surrogate model and the neural network is always calculated based on the test split of the original data.

\subsection{Experimental Results}\label{ssec:exp_results}

\subsubsection{Visual Evaluation for Different Distributions}\label{sssec:visual_eval}
In this experiment, we show the importance of knowing the distribution of the training data in a controlled, synthetic setting. We use a two-dimensional dataset which allows a visual comparison of the decision boundaries (Figure~\ref{fig:graph-eval-combined}). The data used for training the neural networks for this experiment was generated randomly according to Algorithm~\ref{alg:generate-data} but is distinct to the data used for training the \mbox{$\mathcal{I}$-Net} to ensure a fair comparison.

\begin{figure}[tb]

    \includegraphics[width=0.99\columnwidth]{./decision_boundary_graphic_eval_results-combined.pdf}
    \caption{\textbf{Visual Decision Boundary Evaluation.} This figure shows the decision boundaries of the neural network we want to interpret (\RNum{1}), followed by the decision boundary of explanations generated by different approaches, along with their performance for three different datasets and function families. Only when the training data is accessed (\RNum{2}) and when using the \mbox{$\mathcal{I}$-Net} (\RNum{6}), the explanation comprises the relevant aspects of the model. When the training data is not accessible (\RNum{3})-(\RNum{5}), sample-based approaches are not able to generate reasonable explanations.}
    \label{fig:graph-eval-combined}
\end{figure}

Figure~\ref{fig:graph-eval-combined}(a) shows a decision boundary learned by a neural network that ranges from the bottom left corner to the middle right. Thereby, many data points that were assigned to \emph{Class 0} by the neural network are located in the bottom left corner. In contrast, the top right part contains few to no data points.
When the training data is available (\RNum{2}), the standard DT learned a decision boundary that closely matches the decision boundary of the neural network, including the area in the bottom left. However, if the training data is not available, the sample-based approach (\RNum{3}-\RNum{5}) only comprises the large area towards the top and neglects the small area at the bottom left. Considering just the shapes and size of the areas created by the decision boundary, this seems to be a reasonable explanation. However, as explained in Section~\ref{ssec:reasonable_explanations}, if we take the data into account, it becomes apparent that the neglected part of the decision boundary at the bottom left is much more important, since many data points are located in this area.
In contrast, the explanation generated by the \mbox{$\mathcal{I}$-Net} as sample-free approach (\RNum{6}) correctly separates the samples at the bottom left with its decision boundary and neglects the part at the top right, which is not relevant when taking the data into account.
We can confirm this by taking the fidelity scores into account: The \mbox{$\mathcal{I}$-Net} achieved a fidelity of $99.8\%$, while the sample-based distillation without training data only achieved a maximum fidelity of $36.2\%$.

For univariate and standard SDTs we can observe similar results: The fidelity for the sample-based approaches (\RNum{3})-(\RNum{5}) significantly decreased if the training data is not available and the explanation focused on non-relevant areas. In contrast, the \mbox{$\mathcal{I}$-Net} (\RNum{6}) was able to generate high-fidelity explanations without accessing the training data.

\subsubsection{Real World Datasets Performance Comparison}\label{sssec:real_world_eval}
In this experiment, we compare the performance of the \mbox{$\mathcal{I}$-Net} and a sample-based distillation without access to train data on real-world datasets. We selected $8$ commonly used datasets, mostly focusing on the banking and medical domain, comprising personal, confidential data where it is realistic to assume that the training data cannot be exposed.
A description of the datasets, the preprocessing that was performed, and the performance of the neural networks we want to interpret is summarized in Appendix~\ref{A:real-world-data}.

Table~\ref{tab:eval-results-vanilla}-\ref{tab:eval-results-sdt} show the results on the selected datasets for the \mbox{$\mathcal{I}$-Net} and a sample-based distillation using the data generation methods introduced in Section~\ref{ssec:experimental_setup} for the different function families.
For the sample-based distillation, the results show the mean and standard deviation over $10$ trials. While for the standard uniform and standard normal sampling only the sampled data points differ, we sampled a new set of distributions and parameters for each trial in the multi-distribution case.
For a better visual comparison, we highlighted a value bold if it shows a significantly higher fidelity for a certain dataset with $95\%$ confidence according to an unpaired t-test\footnote{For the t-test calculation, we only compared the \mbox{$\mathcal{I}$-Net} with the distillation approach for the sampling strategy with the highest fidelity. Therefore, a bold value only means that a sampling strategy was significantly better than the \mbox{$\mathcal{I}$-Net} and not that it was significantly better than the other sampling strategies. A bold value for the \mbox{$\mathcal{I}$-Net} means that it achieved a significantly higher accuracy than the best sampling strategy for the respective dataset.}.

\paragraph{Standard Decision Trees}
Comparing the results for standard DTs as surrogate model in Table~\ref{tab:eval-results-vanilla}, the \mbox{$\mathcal{I}$-Net} had a significantly higher fidelity on $5/8$ datasets. A sample-based distillation achieved the highest fidelity on $2/8$ datasets.
Even though the multi-distribution sampling strategy did not achieve the best performance on any dataset, it had the highest average performance among the sample-based distillation methods, with a mean fidelity of $73.2\%$. Nevertheless, the performance of the \mbox{$\mathcal{I}$-Net} was significantly better and achieved a mean fidelity of $86.19\%$. Especially for the \emph{Breast Cancer Wisconsin Original} and \emph{Wisconsin Diagnostic Breast Cancer}, the sample-based distillation was not able to generate accurate explanations if the training data was not accessible. For those datasets, the fidelity of sample-based distillation was often even worse than a random guess, which, as already shown in Section~\ref{sssec:visual_eval}, highlights the importance of querying the model on reasonable data points.

\begin{table}[tb]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{l|cccc}
\toprule
\textbf{Dataset} &                                          \mbox{\textbf{\boldsymbol{$\mathcal{I}$}-Net}} & \textbf{Multi-Distribution} &                         \textbf{Standard Uniform} &                                \textbf{Standard Normal} \\
\midrule
\textbf{Titanic (n=9)                            } &  \fontseries{b}\selectfont\phantom{0}95.51 $\pm$ \phantom{0}0.00 &                      \phantom{0}71.12 $\pm$ 17.16 &        \phantom{0}86.07 $\pm$ \phantom{0}3.30 &        \phantom{0}86.29 $\pm$ \phantom{0}7.75 \\
\textbf{Medical Insurance (n=9)                  } &        \phantom{0}82.71 $\pm$ \phantom{0}0.00 &            \phantom{0}88.12 $\pm$ \phantom{0}6.71 &        \phantom{0}89.47 $\pm$ \phantom{0}4.19 &  \fontseries{b}\selectfont\phantom{0}90.75 $\pm$ \phantom{0}8.83 \\
\textbf{Breast Cancer Wisconsin Original (n=9)             } &  \fontseries{b}\selectfont\phantom{0}97.10 $\pm$ \phantom{0}0.00 &                      \phantom{0}83.62 $\pm$ 13.09 &                  \phantom{0}39.42 $\pm$ 13.90 &        \phantom{0}31.88 $\pm$ \phantom{0}0.00 \\
\textbf{Wisconsin Diagnostic Breast Cancer (n=10)} &  \fontseries{b}\selectfont\phantom{0}80.36 $\pm$ \phantom{0}0.00 &                      \phantom{0}56.43 $\pm$ 17.65 &                  \phantom{0}37.86 $\pm$ 15.56 &        \phantom{0}33.39 $\pm$ \phantom{0}5.42 \\
\textbf{Heart Disease (n=13)                     } &        \phantom{0}73.33 $\pm$ \phantom{0}0.00 &            \phantom{0}74.67 $\pm$ \phantom{0}9.45 &  \fontseries{b}\selectfont\phantom{0}85.67 $\pm$ \phantom{0}5.97 &        \phantom{0}80.33 $\pm$ \phantom{0}7.67 \\
\textbf{Cervical Cancer (n=15)                   } &  \fontseries{b}\selectfont\phantom{0}84.71 $\pm$ \phantom{0}0.00 &                      \phantom{0}65.41 $\pm$ 27.77 &        \phantom{0}71.88 $\pm$ \phantom{0}9.64 &                  \phantom{0}60.82 $\pm$ 30.29 \\
\textbf{Loan House (n=16)                        } &                  100.00 $\pm$ \phantom{0}0.00 &                      \phantom{0}77.05 $\pm$ 24.41 &        \phantom{0}96.89 $\pm$ \phantom{0}7.42 &                  \phantom{0}59.84 $\pm$ 33.84 \\
\textbf{Credit Card Default (n=23)                       } &  \fontseries{b}\selectfont\phantom{0}75.80 $\pm$ \phantom{0}0.00 &                      \phantom{0}69.16 $\pm$ 17.58 &        \phantom{0}74.76 $\pm$ \phantom{0}0.05 &                  \phantom{0}34.33 $\pm$ 20.31 \\
\midrule
\textbf{Mean Fidelity                               } &        \phantom{0}86.19 &            \phantom{0}73.20 &                  \phantom{0}72.75  &                  \phantom{0}59.70  \\
\bottomrule
\end{tabular
}
\caption{\textbf{Real-World Evaluation Results for Standard Decision Trees.}}
\label{tab:eval-results-vanilla}
\end{table}

\paragraph{Soft Decision Trees}

For univariate SDTs, the performance difference between \mbox{$\mathcal{I}$-Nets} and sample-based methods is even more pronounced: \mbox{$\mathcal{I}$-Nets} achieved a significantly higher fidelity in $7/8$ cases with a mean fidelity of $82.3\%$, while the best sampling strategy only achieved a mean fidelity of $51.02\%$.

\begin{table}[tb]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{l|cccc}
\toprule
\textbf{Dataset} &                                          \mbox{\textbf{\boldsymbol{$\mathcal{I}$}-Net}} & \textbf{Multi-Distribution} &                         \textbf{Standard Uniform} &                                \textbf{Standard Normal} \\
\midrule
\textbf{Titanic (n=9)                            } &  \fontseries{b}\selectfont\phantom{0}95.51 $\pm$ \phantom{0}0.00 &                      \phantom{0}47.75 $\pm$ 14.70 &            \phantom{0}67.19 $\pm$ 10.50 &        \phantom{0}61.80 $\pm$ \phantom{0}8.35 \\
\textbf{Medical Insurance (n=9)                  } &  \fontseries{b}\selectfont\phantom{0}84.96 $\pm$ \phantom{0}0.00 &                      \phantom{0}56.47 $\pm$ 13.40 &            \phantom{0}48.05 $\pm$ 10.97 &                  \phantom{0}49.62 $\pm$ 10.79 \\
\textbf{Breast Cancer Wisconsin Original (n=9)             } &  \fontseries{b}\selectfont\phantom{0}73.91 $\pm$ \phantom{0}0.00 &            \phantom{0}30.87 $\pm$ \phantom{0}2.25 &  \phantom{0}31.88 $\pm$ \phantom{0}0.00 &        \phantom{0}33.33 $\pm$ \phantom{0}7.45 \\
\textbf{Wisconsin Diagnostic Breast Cancer (n=10)} &  \fontseries{b}\selectfont\phantom{0}82.14 $\pm$ \phantom{0}0.00 &                      \phantom{0}48.57 $\pm$ 22.64 &  \phantom{0}28.57 $\pm$ \phantom{0}0.00 &        \phantom{0}28.57 $\pm$ \phantom{0}0.00 \\
\textbf{Heart Disease (n=13)                     } &  \fontseries{b}\selectfont\phantom{0}63.33 $\pm$ \phantom{0}0.00 &            \phantom{0}60.00 $\pm$ \phantom{0}0.00 &  \phantom{0}60.00 $\pm$ \phantom{0}0.00 &        \phantom{0}60.00 $\pm$ \phantom{0}0.00 \\
\textbf{Cervical Cancer (n=15)                   } &        \phantom{0}83.53 $\pm$ \phantom{0}0.00 &                      \phantom{0}77.06 $\pm$ 20.70 &  \phantom{0}84.59 $\pm$ \phantom{0}0.35 &  \fontseries{b}\selectfont\phantom{0}84.71 $\pm$ \phantom{0}0.00 \\
\textbf{Loan House (n=16)                        } &            \bftab100.00 $\pm$ \phantom{0}0.00 &            \phantom{0}13.11 $\pm$ \phantom{0}0.00 &  \phantom{0}13.11 $\pm$ \phantom{0}0.00 &        \phantom{0}13.11 $\pm$ \phantom{0}0.00 \\
\textbf{Credit Card Default (n=23)                       } &  \fontseries{b}\selectfont\phantom{0}75.03 $\pm$ \phantom{0}0.00 &                      \phantom{0}65.25 $\pm$ 16.47 &  \phantom{0}74.73 $\pm$ \phantom{0}0.00 &                  \phantom{0}65.57 $\pm$ 16.60 \\
\midrule
\textbf{Mean Fidelity                                  } &                  \phantom{0}82.30  &                      \phantom{0}49.89  &            \phantom{0}51.02 &                  \phantom{0}49.59 \\
\bottomrule
\end{tabular
}
\caption{\textbf{Real-World Evaluation Results for Univariate Soft Decision Trees.}}
\label{tab:eval-results-sdt1}
\end{table}

Finally, for standard SDTs, the \mbox{$\mathcal{I}$-Net} achieved a significantly higher fidelity on $3/8$ datasets. The best sampling strategy for SDTs was a standard uniform distribution, which achieved a significantly higher accuracy than the \mbox{$\mathcal{I}$-Net} on $2/8$ datasets. Comparing the mean fidelities, the performance of the \mbox{$\mathcal{I}$-Net} with a fidelity of $92.1\%$ was considerably higher than sampling from a standard uniform distribution ($82.2\%$). This was mainly caused by the superior performance of the \mbox{$\mathcal{I}$-Net} on the \emph{Cervical Cancer} and \emph{Credit Card Default} dataset, where the fidelity was more than $40$ percentage points higher than sampling from standard uniform distribution. \\

\begin{table}[tb]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{l|cccc}
\toprule
\textbf{Dataset} &                                          \mbox{\textbf{\boldsymbol{$\mathcal{I}$}-Net}} & \textbf{Multi-Distribution} &                         \textbf{Standard Uniform} &                                \textbf{Standard Normal} \\
\midrule
\textbf{Titanic (n=9)                            } &  \fontseries{b}\selectfont\phantom{0}95.51 $\pm$ \phantom{0}0.00 &            \phantom{0}88.31 $\pm$ \phantom{0}3.80 &        \phantom{0}92.47 $\pm$ \phantom{0}1.24 &  \phantom{0}92.81 $\pm$ \phantom{0}0.75 \\
\textbf{Medical Insurance (n=9)                  } &        \phantom{0}77.44 $\pm$ \phantom{0}0.00 &                      \phantom{0}79.25 $\pm$ 20.79 &  \fontseries{b}\selectfont\phantom{0}91.20 $\pm$ \phantom{0}7.59 &  \phantom{0}78.50 $\pm$ \phantom{0}0.60 \\
\textbf{Breast Cancer Wisconsin Original (n=9)             } &                  100.00 $\pm$ \phantom{0}0.00 &            \phantom{0}96.67 $\pm$ \phantom{0}4.49 &                  100.00 $\pm$ \phantom{0}0.00 &  \phantom{0}31.88 $\pm$ \phantom{0}0.00 \\
\textbf{Wisconsin Diagnostic Breast Cancer (n=10)} &        \phantom{0}94.64 $\pm$ \phantom{0}0.00 &                      \phantom{0}84.82 $\pm$ 16.98 &  \fontseries{b}\selectfont\phantom{0}97.50 $\pm$ \phantom{0}2.55 &  \phantom{0}28.57 $\pm$ \phantom{0}0.00 \\
\textbf{Heart Disease (n=13)                     } &                  100.00 $\pm$ \phantom{0}0.00 &                      \phantom{0}90.67 $\pm$ 12.45 &        \phantom{0}99.33 $\pm$ \phantom{0}1.33 &  \phantom{0}60.00 $\pm$ \phantom{0}0.00 \\
\textbf{Cervical Cancer (n=15)                   } &  \fontseries{b}\selectfont\phantom{0}85.88 $\pm$ \phantom{0}0.00 &                      \phantom{0}58.35 $\pm$ 21.36 &                  \phantom{0}43.29 $\pm$ 11.54 &  \phantom{0}25.76 $\pm$ \phantom{0}2.38 \\
\textbf{Loan House (n=16)                        } &                  100.00 $\pm$ \phantom{0}0.00 &                      \phantom{0}50.82 $\pm$ 39.36 &                  100.00 $\pm$ \phantom{0}0.00 &            \phantom{0}17.21 $\pm$ 12.30 \\
\textbf{Credit Card Default (n=23)                       } &  \fontseries{b}\selectfont\phantom{0}83.30 $\pm$ \phantom{0}0.00 &                      \phantom{0}59.86 $\pm$ 21.99 &        \phantom{0}38.77 $\pm$ \phantom{0}6.06 &  \phantom{0}75.40 $\pm$ \phantom{0}1.32 \\
\midrule
\textbf{Mean Fidelity                                  } &        \phantom{0}92.10  &                      \phantom{0}76.09  &                  \phantom{0}82.82  &            \phantom{0}51.27 \\
\bottomrule
\end{tabular
}

\caption{\textbf{Real-World Evaluation Results for Standard Soft Decision Trees.}}
\label{tab:eval-results-sdt}
\end{table}

Furthermore, we observed that the average fidelity of standard SDTs is considerably higher than the fidelity of univariate SDTs and standard DTs. We can trace this back to the fact that SDTs have a significantly higher complexity, especially with an increasing number of variables, as shown in Section~\ref{sssec:inets_dts_soft}.
This can also explain why the performance difference between a sample-based distillation and the \mbox{$\mathcal{I}$-Net} is comparatively small for SDTs:
While using meaningful samples for querying the neural network is very crucial when the surrogate model has low complexity, it is less crucial if the surrogate model is more complex, making it less reliant on focusing on the most important information. Accordingly, it is less likely that relevant areas are neglected with an increasing complexity of the surrogate model. However, for interpretability, we are usually interested in surrogate models with a comparatively low complexity that are understandable for humans. In this scenario, $\mathcal{I}$-Nets substantially outperformed sample-based methods. \\

Summed up, the \mbox{$\mathcal{I}$-Net} outperformed a sample-based distillation on the majority of datasets when training data was not accessible for each type of surrogate model considered in this paper. In total, the \mbox{$\mathcal{I}$-Net} achieved a significantly higher fidelity in $15/24$ evaluated cases and only in $5/24$ cases a significantly lower fidelity.
Especially for surrogate models with low complexity, sample-based approaches are dependent on proper querying. Therefore, using the \mbox{$\mathcal{I}$-Net} in such scenarios can achieve a higher fidelity of the surrogate model. This can be crucial since wrong explanations can lead to wrong decisions, as we will evaluate more in-depth in Section~\ref{sssec:case_study_eval}.

\subsubsection{Ablation Study}\label{sssec:improvements_eval}
In Section~\ref{sssec:data_generation} we introduced an improved data generation method which should be more robust in a real-world scenario, since it considers multiple different distributions. In the following, we will compare our new data generation method with the data generation method introduced by \citet{marton2022explanations}, which generates data based on the function family of the surrogate model and considers only a single distribution.
As shown in Table~\ref{tab:eval-results-data-gen}, we were able to outperform the standard data generation method in $7/8$ datasets for standard DTs, $8/8$ datasets for univariate SDT and $6/8$ datasets for the standard SDT.
Summed up, the improved data generation method achieved a higher fidelity in  $21/24$ cases, while the old data generation only achieved a higher fidelity in $1/24$ cases.
Comparing the average performance over all datasets, we also observed a significant increase in the accuracy using the new data generation method of $\approx 36$ percentage points for standard DTs and $\approx 43$ percentage points for univariate SDTs. For standard SDTs, the difference in the mean fidelity was significantly smaller, with only $\approx 6$ percentage points. One explanation could be the higher complexity of the surrogate model for standard STDs, as already discussed in Section~\ref{sssec:real_world_eval}.

\begin{table}[tb]
\small
\centering
\begin{tabular}{l|cc|cc|cc}
\toprule
\multicolumn{1}{c|}{\multirow{2}{*}{\textbf{Dataset}}} & \multicolumn{2}{c|}{\textbf{Standard DT}} & \multicolumn{2}{c|}{\textbf{Univariate SDT}} & \multicolumn{2}{c}{\textbf{Standard SDT}} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{\textbf{new}} & \multicolumn{1}{c|}{\textbf{old}} & \multicolumn{1}{c}{\textbf{new}} & \textbf{old} & \textbf{new} & \textbf{old} \\ \midrule

\textbf{Titanic (n=9)                            } &  \fontseries{b}\selectfont\phantom{0}95.51 & \phantom{0}39.33 &  \fontseries{b}\selectfont\phantom{0}95.51 & \phantom{0}60.67 &  \fontseries{b}\selectfont\phantom{0}95.51 & \phantom{0}86.52 \\
\textbf{Medical Insurance (n=9)                  } &  \fontseries{b}\selectfont\phantom{0}82.71 & \phantom{0}72.93 &  \fontseries{b}\selectfont\phantom{0}84.96 & \phantom{0}77.44 &        \phantom{0}77.44 & \fontseries{b}\selectfont\phantom{0}92.48 \\
\textbf{Breast Cancer Wisconsin Original (n=9)   } &  \fontseries{b}\selectfont\phantom{0}97.10 & \phantom{0}31.88 &  \fontseries{b}\selectfont\phantom{0}73.91 & \phantom{0}31.88 &  \bftab100.00           & \phantom{0}98.55 \\
\textbf{Wisconsin Diagnostic Breast Cancer (n=10)} &  \fontseries{b}\selectfont\phantom{0}80.36 & \phantom{0}28.57 &  \fontseries{b}\selectfont\phantom{0}82.14 & \phantom{0}28.57 &  \fontseries{b}\selectfont\phantom{0}94.64 & \phantom{0}83.93  \\
\textbf{Heart Disease (n=13)                     } &  \fontseries{b}\selectfont\phantom{0}73.33 & \phantom{0}60.00 &  \fontseries{b}\selectfont\phantom{0}63.33 & \phantom{0}60.00 &  \bftab100.00           & \phantom{0}80.00 \\
\textbf{Cervical Cancer (n=15)                   } &  \phantom{0}84.71       & \phantom{0}84.71 &  \fontseries{b}\selectfont\phantom{0}83.53 & \phantom{0}15.29 &  \fontseries{b}\selectfont\phantom{0}85.88 & \phantom{0}84.71 \\
\textbf{Loan House (n=16)                        } &            \bftab100.00 & \phantom{0}13.11 &  \bftab100.00           & \phantom{0}13.11 &        100.00           & 100.00 \\
\textbf{Credit Card Default (n=23)                       } &  \fontseries{b}\selectfont\phantom{0}75.80 & \phantom{0}74.73 &  \fontseries{b}\selectfont\phantom{0}75.03 & \phantom{0}25.27 &  \fontseries{b}\selectfont\phantom{0}83.30 & \phantom{0}64.97 \\
\midrule
\textbf{Mean Fidelity                            } &  \fontseries{b}\selectfont\phantom{0}86.19 & \phantom{0}50.66 &  \fontseries{b}\selectfont\phantom{0}82.30 & \phantom{0}39.03 &  \fontseries{b}\selectfont\phantom{0}92.10 & \phantom{0}86.40 \\
\bottomrule
\end{tabular}
\caption{\textbf{Data Generation Performance Comparison.}}
\label{tab:eval-results-data-gen}
\end{table}

\subsubsection{Case Study: Explaining Neural Networks for Credit Card Default Prediction}\label{sssec:case_study_eval}
In this section, we will take a closer look at the explanations generated by sample-based approaches and the \mbox{$\mathcal{I}$-Net} by returning to \emph{Example~1} which we introduced in Section~\ref{sec:introduction}.
The purpose of this experiment is to show in a real-world setting that without access to the training data, the surrogate model generated by sample-based approaches can lead to incorrect assumptions on the function learned by the neural network.
We want to note that without access to the training data, it is not possible to identify for a specific surrogate model whether it contains a misconception or not, since we are not able to calculate a representative fidelity.

The \emph{Credit Card Default} dataset is concerned with credit card default prediction based on $23$ features including demographic data and the credit history of clients in Taiwan~\citep{dataset_credit_card}. The demographic features include the sex, the education level, the marital status and the age. Furthermore, the repayment status, the amount of the bill and the amount of previous payments, each for the past $6$ months and the credit limit are listed for each client. The dataset is available in the UCI Machine Learning repository~\citep{UCI_ml_repository}, and a more detailed summary of the features with a short explanation is given in Table~\ref{tab:dataset_description}. \\

Figure~\ref{fig:vanilla_credit_card} shows the standard DT surrogate models generated by the \mbox{$\mathcal{I}$-Net} and a sample-based distillation using the multi-distribution sampling strategy.
As shown in Figure~\ref{fig:vanilla_credit_card_inet}, the \mbox{$\mathcal{I}$-Net} archived a fidelity of $75.8\%$ and only considers a single split to decide whether there will be a payment or not. The split is based on whether the payment for the previous month was delayed for less than three months (left path) or not (right path). We can consider this as a very reasonable split, if we assume that if a client was in default previously, there is a higher possibility that there will be a default again. If we take the probabilities at the leafs into account, we can get some more information on the decision process. If the payment for 5 months ago was also delayed less than 4 months, the probability that there will be no default is even higher, as shown in the left branch of the tree. If there was a delay of more than 4 months, the probability that there will be a default is approximately $20\%$ higher. Furthermore, as we can see in the split at the bottom-right, if the past payment amount was higher than $143,000$, the chance of a default is approximately $10\%$ higher in this branch.

In contrast, when taking a closer look at the DT generated by a sample-based distillation (Figure~\ref{fig:vanilla_credit_card_sample}), we can observe that the entire right branch of the tree has \emph{No Default} as prediction. This prediction is made solely based on the first split, where the right branch is taken if the payment amount $6$ months ago was larger than $373,000$. This translates to the rule that we should always predict that there will be no default in the payment if there was a large payment amount in the past. However, it seems counter-intuitive to make this decision without taking for instance the credit history of the client and whether there were defaults previously into account. This is confirmed by the poor fidelity of the surrogate on the real data, which was only $25.3\%$ and worse than a random guess. However, the surrogate model had a very high fidelity of $82.7\%$ on the sampled data used for querying the model which leads to this misconception, since the model appears to have a high fidelity that does not hold on the real data. Without access to the training data, it is not possible to identify this misconception and taking the high fidelity on the sampled data into account, we might assume that the surrogate actually represents what the model has learned and therefore would make wrong assumptions on its behavior.

\begin{figure}[tb]
\centering
\begin{subfigure}{.5\columnwidth}
  \centering
  \includegraphics[width=.99\columnwidth]{./DT_example_inet.pdf}
  \caption{\mbox{$\mathcal{I}$-Net} Standard DT}
  \label{fig:vanilla_credit_card_inet}
\end{subfigure}%
\begin{subfigure}{.5\columnwidth}
  \centering
  \includegraphics[width=.99\columnwidth]{./DT_example_sample.pdf}
  \caption{Sample-Based Standard DT}
  \label{fig:vanilla_credit_card_sample}
\end{subfigure}
\caption{\textbf{Explanation Comparison for Standard Decision Trees.} The surrogate model on the right is learned by a sample-based distillation with a multi-distribution sampling strategy. The DT on the left is predicted by the \mbox{$\mathcal{I}$-Net}. The \mbox{$\mathcal{I}$-Net} makes reasonable splits and achieves a significantly higher fidelity on the real data.}
\label{fig:vanilla_credit_card}
\end{figure}

We can observe similar misconceptions for the function family of univariate and standard SDTs. However, we will not discuss them here in detail, but refer to the Appendix~\ref{A:case_study} for an in-depth evaluation of the explanations generated for those function families.

\section{Related Work}\label{sec:related_work}
Various methods to interpret black-box models have been proposed in the past decades. Overviews from different perspectives are given by \citet{doshi2017towards}, \citet{lipton2018mythos} and \citet{molnar2020interpretable}. In this paper, we focus on methods that translate neural networks into DTs to interpret the underlying decision function.

Model distillation is a common technique to transfer knowledge from a complex model into a surrogate model \citep{bucilua2006model,hinton2015distilling}. It can be used to obtain more compact model representations for efficiency reasons \citep{bucilua2006model,hinton2015distilling,furlanello2018born} or to interpret the model as a human-understandable function \citep{frosst2017distilling,tan2018learning}.
With the focus on interpretability, model distillation is performed to either understand the function encoded by trained networks and how predictions are made \citep{craven1996extracting,boz2000converting,zhang2019interpreting} or to improve the performance of an interpretable algorithm to use it instead of the neural network at test time \citep{krishnan1999extracting,frosst2017distilling,liu2018improving}. Although those purposes differ, the methods can be interchangeably used for both.

Various sample-based methods using DTs as surrogate models were presented in the past quarter-century \citep{craven1996extracting,krishnan1999extracting,boz2000converting,johansson2009evolving,frosst2017distilling,liu2018improving,zhang2019interpreting,nguyen2020towards}.
These approaches have in common that they transform a trained neural network into a surrogate function with a tree-like structure. This is usually achieved by maximizing the fidelity to the neural network on a sample basis.
The main differences among existing approaches are the type of the resulting DTs, the method to train the surrogate model, and the purpose of the surrogate model.

The proposed trees make either univariate \citep{krishnan1999extracting,boz2000converting,liu2018improving}, $m$-of-$n$ \citep{craven1996extracting} or multivariate \citep{nguyen2020towards, frosst2017distilling} decisions at each split.
Trees that consider multiple variables can achieve higher fidelity and accuracy than univariate DTs. However, especially for tabular data, the interpretation becomes harder.

For training the surrogate model, differences exist regarding the data used, the decision how a split is determined, and the optimization technique used.
Regarding the training of trees, most approaches rely on standard DT induction methods. \citet{krishnan1999extracting} use ID3 \citep{quinlan1986induction} and C4.5 \citep{quinlan2014c4}, \citet{craven1996extracting} use ID2-of-3 \citep{murphy1991id2} and \citet{nguyen2020towards} use CART \citep{breiman1984classification}.
While these approaches greedily optimize the fidelity, \citet{frosst2017distilling} use gradient descent to distill the trees.

In the literature, the data to maximize the fidelity is either the training data used for the neural network \citep{frosst2017distilling,liu2018improving} or data from a distribution that was modeled based on the train data \citep{craven1996extracting,krishnan1999extracting,boz2000converting,johansson2009evolving}.
The latter has been claimed to be an effective way to improve the results \citep{boz2000converting,johansson2009evolving}.

In all cases, the predictions on data are the only source for understanding the black-box model behavior, and thus meaningful examples are crucial for the performance.
Without information about the distribution of the training data, e.g., in the form of data points, the performance of sample-based methods decreases significantly.
Recent model distillation approaches deal with this problem using metadata, such as layer activations, to create good samples based on network information \citep{lopes2017data,bhardwaj2019dream,nayak2019zero}.
However, they often still need access to the training data in some part of the distillation process. \citet{lopes2017data} use a fraction of the original training data to compute activations summaries to later compress the network without accessing the data. Similarly, \citet{bhardwaj2019dream} require samples of the original training data to generate activation vectors, which are necessary for their distillation. However, they report requiring significantly fewer data points than \citet{lopes2017data}. In contrast, \citet{nayak2019zero} does not require access to training data, but only requires the model internals. The model internals are used to generate a class similarity matrix based on the parameters of the softmax output layer of the neural network. Based on the class similarity matrix, \citet{nayak2019zero} generate meaningful samples called \emph{Data Impressions} via Dirichlet sampling based on the output classes. However, the approach requires a softmax output for the neural network and is tailored towards multi-class classification problems, since it utilizes the knowledge contained in the class similarity matrix for sampling. Accordingly, an application on a binary classification task is not adequate, since a class similarity matrix for two classes can contain only little information, which makes sampling difficult.
Summed up, the main issue is that the majority of state-of-the-art sample-free approaches still need access to at least a subset of the training data. Only \citet{nayak2019zero} is applicable if no training data is available, but the application is restricted, e.g., to multi-class tasks.

\section{Conclusion and Future Work}\label{sec:conclusion}
In this paper, we proposed a new instance of \emph{Interpretation Networks} (\mbox{$\mathcal{I}$-Nets}) for tree-based surrogate models and an improved data generation method, making \mbox{$\mathcal{I}$-Net} applicable in a real-world scenario.
While traditional approaches generate explanations sample-based and therefore rely on proper querying, \mbox{$\mathcal{I}$-Nets} utilize the model internals, which implicitly contain all relevant information about the network function. Therefore, \mbox{$\mathcal{I}$-Nets} can generate reasonable explanations in scenarios where the training data is not accessible.

Using our new data generation method, we allow the \mbox{$\mathcal{I}$-Net} to learn how to generalize to neural networks trained on different data distributions. Thereby, the \mbox{$\mathcal{I}$-Net} identifies which aspects learned by the neural network are most important based on the distribution of the training data and therefore should be contained in the explanation. The \mbox{$\mathcal{I}$-Net} can use this knowledge to generate meaningful explanations for new, unseen networks, even without access to the training data.

In our experiments, we showed that sample-based approaches strongly rely on proper querying and are often not able to generate reasonable explanations once they have no access to the training data. In this scenario, the explanations of sample-based approaches frequently comprise misconceptions, since they focus on the global behavior and do not focus on the regions that are important for a reasonable explanation, as we demonstrated within our case study.
Furthermore, we empirically showed that \mbox{$\mathcal{I}$-Nets} consistently outperform sample-based methods on real-world datasets when the training data is not available.
Thus, using \mbox{$\mathcal{I}$-Nets}, high-fidelity explanations can be generated when confidential training data can't be exposed.

Currently, the \mbox{$\mathcal{I}$-Net} comprises a feed-forward neural network and the model internals used as an input are flattened to a one-dimensional array. In further work, we aim for a more sophisticated \mbox{$\mathcal{I}$-Net} architecture and a better-suited representation for the model input to improve the performance even further.
Furthermore, the \mbox{$\mathcal{I}$-Net} is tailored towards generating fully grown DTs based on its structure. In further work this could be addressed by adjusting the output layer which allows using greater depths for the explanation without a significant increase in complexity.

redpajama_set_name: RedPajamaArXiv


• A family friendly environment where people can come to create their own masterpieces. Paint a Piece is where "Everyone's an artist!" place for birthday parties, private parties, custom painting, summer camps and much more.
• We love to help our customers find their "inner artist" even when they think there isn't one. That means being a friendly, helpful and knowledgeable employee at Paint a Piece.
• A tight-knit group that must work as a team. We work hard but we have fun doing it.
• A place to learn: we will teach you how to work with paints, teach techniques, host birthday parties and summer camps, build with clay, fuse glass, glaze pottery and load kilns.
• An equal opportunity employer.
• Be responsible for opening or closing the store without supervision.
• Have great verbal communication skills. Must be comfortable talking to strangers and large groups of adults and children.
• Have good spelling and writtern communication skills.
• This is a family friendly environment. Our employees present a drug- and alcohol-free appearance, attitude and reputation at all times.
This position entails helping customers, set-up and clean-up, working as a team, and taking initiative to make our studio a better place for our customers.
• Greeting customers and teaching them about our wide variety of products. Employees are here to inspire, guide, teach, and clean up after our customers. It is much like being a waitress.
• Mopping, dusting, taking out the trash, and cleaning tables, chairs, counters, windows, and more. Maintaining a clean and inviting studio for our customers is a major priority.
Do you love being creative? Do you love working with kids? Want to get paid to do something you love?


I'm as shocked at the notion as you probably are. My recent loving ode to the Phoenix Suns was picked up by the team's webmaster and added to their Twitter stream. Within no time at all, I received an e-mail from the senior content editor at Fanster – a truly comprehensive sports website dedicated to covering all professional and college-level sports in Arizona. It seemed as though they were interested in some north-of-the-border perspective on the Suns and offered me the opportunity to become a regular blogger.
I humbly and excitedly accepted.
My first column installment was published this morning (found here) and is more or less an introduction to yours truly. I can't muscle my way into a sports fans' site and pretend I have a clue as to what I'm talking about without some credibility. As it stands now, I'll be publishing my thoughts and ideas each Monday – and possibly more if I feel so inspired. Keep in the loop with my rants and raves by viewing my Fanster profile which will include a listing of all posts.
Congrats! You know that even though I am not a sports fan of any kind that I will be reading your column every week!
That's awesome Keira! Congrats! Hopefully they can fly you out to a game or two!


text:
Immigrant Venezuelan Woman's Unborn Baby Dies While She Waits for Visa, Chile Refuses Medical Aid – News18
Chile's foreign ministry said in a statement that it regretted the death of the unborn baby, but said 'it was not possible to lend medical assistance to foreign citizens in a neighboring country.'
Updated:July 14, 2019, 10:19 AM IST
Representational picture. (AFP)
Santiago: Chilean officials said on Saturday that the unborn baby of a Venezuelan migrant had died while the woman waited for the appropriate visa to cross the border from Peru into Chile.
Peru last month clamped down on immigration, requiring that migrants have passports and visas in order to stay in the country. The new rules have pushed some migrants south to Chile.
Chile's foreign ministry said in a statement that it regretted the death of the unborn baby, but said "it was not possible to lend medical assistance to foreign citizens in a neighboring country."
Venezuela's economic collapse has unleashed the biggest migratory crisis in recent South American history. Chile, one of Latin America's strongest and most stable economies, has become a magnet for Venezuelans seeking a new home.
Immigration into Chile has increased sixfold in less than 30 years, from 114,500 in the 1992 census, to 746,465 last year. The majority of migrants recently have come from Haiti and Venezuela.
Chile's foreign ministry said it had beefed up staff at its consulate in Tacna, Peru, to handle the influx. The consulate has issued more than a thousand visas for Venzuelans seeking entry into the country in the past three weeks, the ministry said.
Previous Article A Registered Dietitian Explains How A Keto Diet Could Boost Fertility – mindbodygreen.com
Next Article Medical examiner: Woman struck by vehicle, killed near Edgerton and Forest Home in Hales Corners – WITI FOX 6 Milwaukee

redpajama_set_name: RedPajamaCommonCrawl


text:
Ford Fiesta Celebrates 40 Years Of Production
Posted on 18th July 2016 21st August 2019 by Perrys Content Team
When Ford launched the Fiesta, 40 years ago in 1976; few could have predicted the phenomenal success story that was going to unfold over the coming decades.
Four Decades
In the 40 years from 1976-2016, Ford sold 4,339,149 Fiestas in the UK alone, with European Fiesta sales set to surpass 18 million before the end of this year. Over these four decades, more than one in every 20 cars sold in the UK was a Ford Fiesta!
In 2014, the Ford Fiesta became the UK's best-selling car of all time, overtaking the Ford Escort. It is not only the best-selling car in the UK this year; it has been the UK's most popular car every month for the last seven years.
To celebrate this success, and the range of vehicles produced over the years, 40 of the nation's favourite Fiestas travelled in convoy from the Ford Engine Plant in Dagenham to Brighton Race Course for the 2016 Ford Summer Festival – A celebration of all Ford models.
The Dagenham to Brighton drive was open to all Ford vehicles; each receiving a unique numbered plaque. Around 200 Fords took part with models ranging from Cortinas to Rangers, and to mark the occasion, numbers 1-40 were allocated to Fiestas from the past four decades.
The Ford Fiesta MkI began production in the all-new Valencia factory in 1976 and sales in the UK commenced in 1977. Production versions of the Ford Fiesta MkI took part in the Dagenham to Brighton drive on Saturday. Ford of Britain chairman and managing director, Andy Barratt, said, "The opportunity to have these wonderful heritage vehicles lined up next to our latest Fiestas, such as the ST200, is very rare and was fully appreciated by the Ford fans that made the 69-mile journey down to the festival."
Also on display at Dagenham was a MkIV 1.4 Fiesta Ghia; the 10 millionth Ford vehicle produced at Dagenham as well as the MkVI World Rally Championship car. The Fiesta MkI owners club were also involved in the drive, bringing with them a variety of sport Fiesta models as well as a MkII Fiesta Popular and a MkI Fiesta Van.
The chronological evolution of Fiesta was evident, from the early MkI to the latest MkVII, and the changes significant, both visually and technologically. External design changes have been accompanied by new technologies, safety features and increased levels of equipment, many of which are now standard on the latest Fiesta, including air-conditioning, electric windows, Easy Fuel, electrically-operated and heated door mirrors and power-assisted steering. Today's Fiesta, with a range of class-leading powertrains – including the multi-award winning 1.0-litre EcoBoost engine, can be ordered with heated seats, cruise control and integrated navigation system which can even include a rear-view camera.
This entry was posted in Ford, News Archives and tagged 40th Annivesrary, Brighton Race Course, Ford, Ford Fiesta, Ford Summer Festival.
Vauxhall Performs Well In Dependability Stakes
Nissan Reveals Its Crossovers Have Tough UV Protection


text:
#include "Diff.h"

#include "GitRepository.h"

#include "../simple/Parser.h"

namespace FilePersistence {

Diff::Diff(){}

Diff::Diff(QList<GenericNode*>& nodesA, std::shared_ptr<GenericTree> treeA,
			  QList<GenericNode*>& nodesB, std::shared_ptr<GenericTree> treeB,
			  const GitRepository*)
{
	treeA_ = treeA;
	treeB_ = treeB;

	IdToGenericNodeHash nodesAHash;

	for (auto node : nodesA)
		nodesAHash.insertMulti(node->id(), node);

	IdToGenericNodeHash nodesBHash;
	for (auto node : nodesB)
		nodesBHash.insertMulti(node->id(), node);

	filterPersistenceUnits(nodesAHash);
	filterPersistenceUnits(nodesBHash);

	computeChanges(nodesAHash, nodesBHash);
	computeStructChanges();
}

IdToChangeDescriptionHash Diff::changes(ChangeType type) const
{
	IdToChangeDescriptionHash changesOfType;
	for (auto change : changeDescriptions_.values())
	{
		if (change->type() == type)
			changesOfType.insert(change->nodeId(), change);
	}
	return changesOfType;
}

IdToChangeDescriptionHash Diff::changes(ChangeType type, ChangeDescription::UpdateFlags flags) const
{
	IdToChangeDescriptionHash changesOfType;
	for (auto change : changeDescriptions_.values())
	{
		if (change->type() == type && change->hasFlags(flags))
			changesOfType.insert(change->nodeId(), change);
	}
	return changesOfType;
}

// Private methods
void Diff::computeChanges(IdToGenericNodeHash& nodesA, IdToGenericNodeHash& nodesB)
{
	QSet<Model::NodeIdType> onlyInNodesB = QSet<Model::NodeIdType>::fromList(nodesB.keys());
	IdToGenericNodeHash::iterator iter;
	for (auto nodeA : nodesA.values())
	{
		iter = nodesB.find(nodeA->id());
		if (iter == nodesB.end())
		{
			// no such id in nodesB
			auto change = std::make_shared<ChangeDescription>(nodeA, nullptr);
			changeDescriptions_.insert(nodeA->id(), change);
		}
		else
		{
			// found id in nodesB
			auto change = std::make_shared<ChangeDescription>(nodeA, iter.value());
			changeDescriptions_.insert(nodeA->id(), change);
			onlyInNodesB.remove(iter.key());
		}
	}
	/* Intermediate state 2
	 * See report for details.
	 * TODO add link to report
	 */
	for (auto id : onlyInNodesB)
	{
		iter = nodesB.find(id);
		auto change = std::make_shared<ChangeDescription>(nullptr, iter.value());
		changeDescriptions_.insert(id, change);
	}
	for (auto change : changeDescriptions_.values())
		if (change->isFake())
			changeDescriptions_.remove(change->nodeId());
}

void Diff::computeStructChanges()
{
	for (auto change : changeDescriptions_.values())
	{
		if (change->type() == ChangeType::Insertion || change->type() == ChangeType::Move)
			setStructureFlagForId(change->nodeB()->parentId(), change);
		if (change->type() == ChangeType::Deletion || change->type() == ChangeType::Move
			 || change->hasFlags(ChangeDescription::Label))
			setStructureFlagForId(change->nodeA()->parentId(), change);
	}
}

void Diff::setStructureFlagForId(Model::NodeIdType id, std::shared_ptr<ChangeDescription> causingChange)
{
	if (!changeDescriptions_.contains(id))
	{
		auto change = ChangeDescription::newStructChange(id, causingChange, treeA_, treeB_);
		changeDescriptions_.insert(id, change);
		change->setStructureChangeFlag(true);
	}
	else
	{
		changeDescriptions_.value(id).get()->setStructureChangeFlag(true);
	}

	Q_ASSERT(changeDescriptions_.find(id).value()->hasFlags(ChangeDescription::Structure));
	Q_ASSERT(changeDescriptions_.value(id)->debugHasNodes());
}

void Diff::filterPersistenceUnits(IdToGenericNodeHash& nodes)
{
	for (auto node : nodes.values())
	{
		if (node->type() == GenericNode::PERSISTENT_UNIT_TYPE)
			nodes.remove(node->id(), node);
	}
}

} /* namespace FilePersistence */

redpajama_set_name: RedPajamaGithub


The owners of INI Farms are an eclectic mix of highly successful global entrepreneurs, impact investors and professional managers. Ronnie Screwvala (UTV/ Disney), Aavishkaar, Ashish Gupta (Junglee, Amazon, Helion Ventures) and Pavan Vaish (IBM Daksh) bring in experience of creating entirely new industries in India and actively provide direction to the business.
Promoted by first generation entrepreneur Ronnie Screwvala, Unilazer Ventures has Treasury Operations, including Fixed Income and Debt Instruments. It is an active equity Investor in the public markets space and commodities market.
Unilazer is also a strategic investor in start ups, early stage and mature companies building an active business model in certain sectors with a team which has unique strength in starting and running businesses and creating businesses with scale, brand and shareholder-value.
Unilazer is the key supporter of the SWADES foundation, which is committed to one of the largest social initiatives in rural Maharashtra.
Aavishkaar is a pioneer in early stage investing in India and has been active in the space for over a decade. They are guided by the fundamental belief that investing in early stage entrepreneurial ventures can not only deliver commercial returns, but also bring about significant efficiencies and developmental impact to rural and underserved communities.
Over time, Aavishkaar has built a track record of high impact scalable enterprises in its portfolio that span across seven key sectors, namely Agriculture and Dairy, Education, Energy, Handicrafts, Health, Water and Sanitation, Technology for Development and Microfinance and Financial Inclusion.
Pavan is the Global Chief Operating Officer managing UnitedLex's business objectives in all geographies. He also serves on UnitedLex's Board of Directors. A visionary among BPO industry leaders, he co-founded Daksh in 2000 which was subsequently acquired by IBM. Pavan has served as the CEO of IBM, Daksh for 11 years, providing strategic guidance and leadership to all client-facing business units and the shared services.
Prior to IBM and the founding of Daksh (which became IBM Global Process Services), Pavan co-founded Quadrant Infotech, an offshore data capturing business focused on the US market.
Pavan holds a Bachelors degree in Mechanical Engineering from Indian Institute of Technology (IIT), Kharagpur and Master's degree from Stanford University. He is also a Graduate of the Advanced Management Program of the Harvard Business School.
Dr. Ashish Gupta is a co-founder of Helion and serves on the boards of NGPay, Exclusively.in, redBus, Dhingana, Komli, Pubmatic, Jivox, Kirusa, MuSigma, Naukri.com, SMSGupshup and Hoopos.com. He has co-founded two successful companies - Tavant Technologies and Junglee (AMZN).
He has also worked at Woodside Fund, Oracle Corporation and IBM research. His investments include Daksh (IBM), Odesk, Obongo (AOL), Speedera (Akamai), MakeMyTrip, Merittrac (Manipal Group), and Kaboodle (Hearst).
Sudhir is the Managing Director of Suntera Energy, an oil and gas company with a focus on operated exploration and development assets in Nigeria / Africa. Sudhir is also the Director of Energy Investments for SUN Group (http://www.sungroup-global.com), a privately held principal investor and private equity fund manager operating in selected emerging markets. Previously, Sudhir worked as a management consultant for McKinsey & Company.
Sudhir holds a Bachelors degree in Computer Science from the Delhi Institute of Technology, University of Delhi and is a Post Graduate from the Indian Institute of Management, Ahmedabad.
Aspada provides early stage risk capital to driven entrepreneurs building innovative, full-stack businesses in India that solve hard problems. They look for disruptive, technology-led solutions that have the ability to organize traditional and fragmented markets.
They support outstanding teams building category leaders in large markets, with the intention of mobilizing significant strategic support and capital as our portfolio companies grow.
They invest out of two vehicles: The Aspada Investment Company is a permanent capital vehicle anchored with a significant commitment from the Soros Economic Development Fund. The SONG Fund is an early stage venture capital fund capitalized by the Soros Economic Development Fund, the Omidyar Network, and Google.


text:
Q: Using super+space rotates through apps in bottom I use super+space to switch languages, but recently it started showing me the app previews (or whatever they are called) in the bottom panel

How can I toggle this off and only use the shortcut for switching languages?
Using Ubuntu 20.04 with cinnamon environment

A: Right click on the Open Applications icon and click Parameters

Click on Settings

On the hotkey tab, look for Global hotkey for cycling through the thumbnail menu

Sorry for not finding the English version.

redpajama_set_name: RedPajamaStackExchange


Welcome to the Comfort Inn located off I-70. Just minutes from Morgan Community College, Bonny Reservoir, Kit Carson County Airport, Old Town Outdoor Museum and several other local attractions and business. The Comfort Inn offers FREE wireless high speed internet, hair dryers, iron/ironing boards, coffee makers and cable TV in all guestrooms. Enjoy our hot breakfast each morning. Take a swim in our indoor heated pool or relax in the hot tub. Microwave, Refrigerator rooms and Suites available. While traveling on business or pleasure the staff at the Comfort Inn will make you feel at home during your stay. Pet Accommodation: 15.00/night. Pet Deposit: 50.00/stay. Pet limit: 2 pets per room.
Blending tranquil ambience, comfortable accommodation and friendly service, this property is an ideal place for business as well as leisure travellers.LocationNestled on the Eastern Plains of Colorado, the Comfort Inn Burlington is few minutes away from the Old Town outdoor museum, the historic Kit Carson County Carousel, Bonny Reservoir and Morgan Community College, while the Kit Carson County Airport is 8 kilometres away. RoomsThis hotel offers immaculate accommodation units that are endowed with all the essential amenities to assure you immense pleasure throughout your stay. RestaurantGuests can begin their day with a delicious Continental breakfast that is served in a warm and pleasant atmosphere of the breakfast room. GeneralFor your business needs, the hotel offers a well-equipped meeting room, capable to accommodate up to 25 people. The on-site recreational facilities include a well-equipped gymnasium, swimming pool as well as a whirlpool.
Burlington hotel with an indoor pool:This Burlington hotel is located near Elitch Gardens Carousel. :Recreational amenities at Comfort Inn Burlington include an indoor pool and a health club. Other amenities include complimentary parking and self parking.
The hot tub and swimming pool were very nice.
The breakfast was excellent, with bacon and french toast among other things.
Not a whole lot to do in this area, but very comfortable and great breakfast.
The beds were a little springy and not very soft.
Very nice hotel, great room, comfortable beds, friendly staff.
The motel itself was very clean and inviting.
I would recommend this hotel to anyone, I stay in a lot of hotels all over the country and this one was great.
They served free meals all day long each day.
Nice clean well organized hotel.
The front-desk staff was very friendly and accommodating.
Clean property, nice staff...Staff was very friendly and organized.