text:
\section{Introduction}
The state of the art convolutional networks has been quite successful at various computer vision tasks. However, the improved performance has come at the cost of reduced human understanding of the model. It is crucial to bring transparency in these networks to help understand their decisions. Recently, there has been a lot of work on designing models that explain the behavior of a pre-trained convolutional network. These approaches generate saliency maps for explaining the model's behavior or explain the prediction based on training instances. Approaches such as GradCAM \cite{gradCAM} and its variants, use gradient information to generate saliency maps as illustrated in Figure \ref{fig:comparison}. Other approaches like Excitation Backpropagation \cite{DBLP:journals/corr/Zhang0BSS16} use top-down neural attention to generate attention maps. These explanations provide a high-level insight into the working of the model. However, we observe that invariably almost the entire object is highlighted in all the saliency maps and their explanations are almost identical for the predicted class and any other class for a given image. This reduces the interpretability of their explanations. While the explanations can detect the foreground object, they are unable to accurately highlight regions in the object that contributed towards the prediction.

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth, height=\linewidth, keepaspectratio]{images/comp/draft_f.png}
\caption{[Best viewed in color] Explanations generated by different approaches for the predicted class and another class for an image of wolf.}
\label{fig:comparison}
\end{figure}
Another style of reasoning for a prediction is by dissecting an image and referring to the presence of smaller concepts. For example, if the image is that of a lion, the smaller concepts could be legs, ears, face, skin texture, etc. The aggregation of these smaller concepts is used to explain the final output. The fundamental objective of our framework: Model-Agnostic Concept Extractor (MACE) is to mimic this style of reasoning; which is to explain the model's behavior in terms of smaller concepts in the image. Our approach learns multiple concepts for a class without any supervision or additional information. These concepts are visualized through multiple high quality localized saliency maps, rather than a single region. Figure \ref{fig:outputIllustration} shows the visualizations of some of the concepts learned by our approach.

\begin{figure}[H]
    \centering
\begin{subfigure}{0.49\textwidth}
  \includegraphics[width=0.24\linewidth]{images/stability/0_5_0_m_.png}%
  \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/0_5_1_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/0_5_4_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/0_5_6_m_.png}
  \caption{Muzzle in class Fox.}
  \label{fig:muzzle}
\end{subfigure}
\hfill
\begin{subfigure}{0.49\textwidth}
  \includegraphics[width=0.24\linewidth]{images/stability/1_9_0_m_.png}%
  \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/1_9_2_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/1_9_4_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/1_9_8_m_.png}
  \caption{Face in class German Shepherd.}
  \label{fig:face}
\end{subfigure}
\\
\begin{subfigure}{0.49\textwidth}
  \includegraphics[width=0.24\linewidth]{images/stability/2_8_5_m_.png}%
  \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/2_8_7_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/2_8_8_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/2_8_9_m_.png}
  \caption{Legs in class Horse.}
  \label{fig:legs}
\end{subfigure}
\hfill
\begin{subfigure}{0.49\textwidth}
  \includegraphics[width=0.24\linewidth]{images/stability/3_2_0_m_.png}%
  \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/3_2_1_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/3_2_6_m_.png}%
    \hfill
  \includegraphics[width=0.24\linewidth]{images/stability/3_2_9_m_.png}
  \caption{Body texture in class Leopard.}
  \label{fig:body}
\end{subfigure}
\caption{[Best viewed in color] Visualization of concepts across multiple images of the same class.}
\label{fig:outputIllustration}
\end{figure}

Further, the proposed framework also estimates the relevance of each concept with respect to the output of the model. This is particularly helpful when the explanation for any output of the model is desired. In Figure \ref{fig:comparison} the first row corresponds to the explanations from different approaches for the predicted class and the second row presents the explanations for the class with second-highest prediction probability. It can be noticed that there is no significant difference in the explanations generated by current approaches for the two outputs of the model. However, our approach can generate class-specific concepts as well as their relevance for a more comprehensive explanation. For an incorrect class, most of the concepts have negative relevance thereby suppressing the prediction probability. As illustrated in Figure \ref{fig:comparison} while the predicted class is the wolf, the second-highest probability is for class fox. The MACE framework extracts only a single concept (that of legs) with positive relevance for the class fox.

We also ensure that the MACE framework generates explanations that satisfy a few desirable properties. \textit{Stability}: The learned concepts should be consistent i.e. the localized saliency maps generated for a particular concept should be similar across all the images of the same class, thus achieving higher interpretability. This can be observed in Figure \ref{fig:outputIllustration} where the visualizations for a single concept obtained from different images appear very similar. \textit{Faithfulness}: The explanations generated by the framework should accurately represent the working of the pre-trained model. This will improve the trustworthiness of the explanations. Finally, \textit{Robustness}: The explanations should be robust to perturbations such as noise, translation, and rotation in the input. Explanations of perturbed inputs should not change significantly if there is no significant change in the corresponding outputs.

\section{Related Work}
The last few years have seen significant work on developing post-hoc explanation models \cite{gradCAM, grad_cam_plus_plus, cam, lime, anchors, shap}. Some of the popular approaches construct saliency maps that highlight important regions in the image. The Class Activation Map (CAM) \cite{cam} is one of the earliest approaches to construct a saliency map specific to a classification model. The gradient-class activation maps (Grad-CAM) \cite{gradCAM} is a generalization of CAM that uses gradients flowing into the final convolutional layer to localize salient image regions. GradCAM is among the most popular approach for generating explanations for an image classification network. However, it has multiple drawbacks including poor localization in the presence of multiple instances of the same object in an image and poor distinction of the class-specific saliency maps generated for different classes, given an input \cite{similar_saliency_ieee}. GradCAM++ \cite{grad_cam_plus_plus}, a variant of GradCAM, overcomes the limitation of generating accurate saliency maps for multiple instances of an object in an image. However, GradCAM++ requires the estimation of higher-order derivatives, the cost of which grows with increasing network complexity.

Vis-CNN \cite{vis_cnn} also uses gradients to explain the contribution of each input pixel towards the output. But pixel-wise quantification appears like a foreground detector looking at most of the results presented in the paper and also during our experiments. Full Grad \cite{full_gradient} combines the idea of using gradients flowing back to intermediary layers as well as gradients flowing back till input to generate a saliency map attributing importance of each pixel towards a prediction. On the other hand, Wang et al \cite{similar_saliency_ieee} suggest a possible compromise in the faithfulness of explanations when gradients are used to generate the explanations. There have been other variants of CAM like Ablation-CAM \cite{ablation_cam} and Score-CAM \cite{score_cam} that avoid the need for gradients for computing the saliency maps. Our approach also estimates saliency maps for explaining the pre-trained model without utilizing the gradient information. Further, we automatically synthesize multiple maps each characterizing a different concept present in the image.

Fong et al, \cite{perturbation_deletion} propose a perturbation based approach for estimating the salient regions in the image, that was further extended to also take into account when the region was preserved \cite{perturbation_preservation}. These complementary approaches aim to modify the input image successively and map the region integral for the final prediction. There are many challenges with the feasibility of the approach given the high dimensional nature of the images and voluminous amount of possible perturbations. Excitation backpropagation \cite{DBLP:journals/corr/Zhang0BSS16} proposes a probabilistic attention mechanism to explain the working of black boxes in a fine-grained manner. But as recent works like \cite{attention_mohankumar_2020,attention_ieee_xu,grimsley2020attention} state, attention need not be human interpretable .

Another category of explainable models is the ante-hoc models that are explainable by design \cite{DLFCBP,this_looks_like_that,hierarchical_prototypes,concept_whitening}. Due to explainability being a parameter considered in the design, the explanations are faithful to the underlying model. However, the need to retrain the model from scratch to incorporate the explainability aspect is an obstacle to explain a pre-trained model that has already been deployed.

Our proposed approach is a posthoc explainability technique like \cite{cam,gradCAM,grad_cam_plus_plus} that leverages some aspects from ante-hoc explainability techniques like \cite{this_looks_like_that, DLFCBP} to generate fine-grained and faithful explanations. Like GradCAM, our approach generates human interpretable explanations in the form of a saliency map. However, unlike GradCAM, we do not utilize the gradient backflow information for generating the saliency maps. Our work is closely related to the exciting paradigm of `this looks like that' proposed by Chen et al \cite{this_looks_like_that}. Specifically, the MACE framework generates multiple saliency maps containing different parts/concepts for explaining the output of a single image. However, unlike \cite{this_looks_like_that}, our approach does not require modifying and retraining the black box architecture nor is there a need to slide across the activation maps or tune additional parameters to visualize a concept.

\section{Methodology}
The MACE framework is envisioned as a modular network that can be attached to any convolution layer of a pre-trained network for probing its functionality. However, for describing the framework, we assume that the MACE network is inserted in between the final convolutional layer and the first dense layer of a pre-trained network. This lateral connection taps into the spatial distribution of concepts (as gathered from the downstream convolutional layers) and the discriminative features for classification (from the upstream dense layers). Given, a query image and a class label, the MACE framework extracts the class-specific low-level concepts along with their relevance towards the output of the model.

The process of learning and extracting the concepts and their relevance is divided across four modules namely; the map generator, embedding generator, relevance estimator, and output generator. The MACE framework is illustrated in Figure \ref{fig:architecture}. The map generator learns an attended spatial map representing the spread of a concept using the output of the last convolutional layer of the pre-trained model. The embedding generator transforms the manifestation of a concept into a latent representation that is invariant to spatial information. The relevance estimator determines the importance of a class-specific concept towards the output of the model. Finally, the output generator completes the loop by linking the extracted concepts back to the first dense layer of the pre-trained model.

Let $f$ be a pre-trained model for a $K$-way image classification task; trained using a dataset $\mathcal{D}$. The output of the last convolutional layer in $f$ for a given input is denoted as $\textbf{x}$. Thus $\textbf{x}\in \mathbb{R}^{H\times W\times D}$, where $H$ and $W$ refer to the size of the feature map and $D$ refers to the number of filters in the last convolution operation. Let $\textbf{z}\in \mathbb{R}^L$ represent the output of the dense layer following the convolutional layer. The MACE framework is laterally connected in between the transformation of $\textbf{x}$ to $\textbf{z}$.

For the sake of simplicity, we assume that every class can be explained by the same number of concepts. (In practice as discussed in the supplementary material section S2, pruning may result in a varying number of concepts per class). A concept has two aspects to it. The first is its actual manifestation in an image, termed as the concept map, for example, the region corresponding to the ear or legs. The second aspect is the latent representation, termed as the concept embedding, that is invariant to the manifestation.  Irrespective of the location and orientation of the concept ear in an image, the encoding of the concept remains the same. $\textbf{c}_{jk} \in \mathbb{R}^{H\times W}$ denotes the $j^{th}$ concept map for the $k^{th}$ class and $\textbf{e}_{jk} \in \mathbb{R}^Q $ denotes the corresponding concept embedding.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{archi.png}
    \caption{[Best viewed in color] Architecture of the MACE Framework.}
    \label{fig:architecture}
\end{figure}

\subsection{Map Generator}
The map generator takes as input the convolutional feature map $\textbf{x}$ and estimates a concept map $\textbf{c}_{jk}$. The concept map encodes two properties - the activation pattern of a concept and the salient region in the image that expresses the activation pattern. The activation pattern is used by the embedding generator to extract a distinct and invariant encoding for the concept. The salient region in the image is used for visualizing the concept post-training, facilitating the human interpretability of the concept.

Due to the nature of convolution operations, the activations of a concept are distributed across the different channels of the feature map $\textbf{x}$. The MACE framework combines the channels in the feature map to produce a single activation pattern for a concept. This is achieved as a weighted average of the channels in the feature map. Specifically, we employ 1D convolutions to obtain the concept map. Let $\theta_{jk}^M$ denote the weights of the 1D convolution filter for generating the $j^{th}$ concept map of the $k^{th}$ class, then $  \textbf{c}_{jk} = ReLU(\textbf{x} * \theta_{jk}^M) $. The $ReLU$ operation is performed because, the locations in the activation map corresponding to large positive values may denote the presence of the concept \cite{gradCAM,grad_cam_plus_plus}. We have to learn $K\times C$ 1D convolution filters, assuming $C$ concepts per class. The concept maps $\textbf{c}_{jk}$ are of the same size as the input feature map $\textbf{x}$.

Each of these concept maps represents an activation pattern. The region in the concept map exhibiting the highest activation is most likely to have expressed the concept. This region in the original image contributing to this activation can be visualized by resizing the concept map to the original image dimension. We use this resized concept map to visualize the concept. The MACE framework does not impose any constraints on the size of the concept. Thus allowing it to learn small concepts such as ears to large concepts like body and background.

\subsection{Embedding Generator}
The next module - the embedding generator takes as input the concept maps that are stacked class-wise. The map generator only uses the convolution layer activation maps to generate the concept map and thus retains the spatial information about the location of the concept in the image. Consider two images one in which the object lion is present on the right half of the image, and a different orientation of the object lion is present in the left half of the second image. The concept maps of these two images will have high activations in different regions even though the underlying concept is the same. Thus, we need to abstract out the concept from its actual manifestation. We refer to this encoding as the concept embedding. We would like the embeddings for a concept extracted from different images belonging to the same class exhibiting the concept to form a tight cluster.

The abstraction of the concept through the embedding also enables a comparison of different concepts. We model the embedding generator as a multi-layer dense network. Having a sufficiently large network enables the extraction of different concepts. However, the parameters of the network are shared across the concepts of a class to reduce the overall learning complexity. Formally, let the embedding generator network for class $k$, be parameterized by $\theta^E_k$. A stack of the concept maps $\{\textbf{c}_{jk}\}_{j=1}^C$ extracted from an input image are fed as inputs to embedding generator resulting in the set of embeddings $\{\textbf{e}_{jk}\}_{j=1}^C$. Restricting to only a positive subspace (as was the case with the concept map generation) is not necessary for learning the concept embedding. We use tanh activations enabling the use of the full space to learn well-separated embeddings.

Triplet loss is used to learn the invariant concept embeddings. We normalize the embeddings before applying the triplet loss. We use the training images belonging to class $k$  from a mini-batch to learn the $C$ concept embeddings for the class. The embeddings generated for concept $j$ across the training images of a particular class are chosen as anchor positives. The embeddings for the other concepts (except $j$) for a specific training image are the anchor negatives. Similar to \cite{schroff2015facenet}, we use all anchor-positive pairs and select semi-hard negatives for anchor-negative pairs. The margin $\alpha$ is set to 1 to make the embeddings orthogonal to each other. Let $\textbf{e}_{jk}^a(i)$ represent an anchor for the $i^{th}$ image in the mini-batch, the anchor positives are represented as $\textbf{e}_{jk}^p(i)$ and the anchor negatives as $\textbf{e}_{jk}^n(i)$. Then the triplet loss for class $k$ using a mini-batch of $B$ images is defined as
\begin{equation}
\mathcal{L}_k^E = \sum_{i=1}^B\sum_{j = 1}^{C}\bigg[ \| \textbf{e}_{jk}^{a}(i) - \textbf{e}_{jk}^{p}(i)\|_{2}^{2} - \|\textbf{e}_{jk}^{a}(i) - \textbf{e}_{jk}^{n}(i)\|_{2}^{2} + \alpha\bigg]_{+}
\label{eq:tripletloss}
\end{equation}

\subsection{Relevance Estimator}
A key aspect of the MACE framework is the estimation of the relevance of the concept towards the model prediction.  This score, termed as concept relevance, quantifies the contribution of a concept towards the output of the model for a given class. The concept relevance would enable a comparison of different concepts for a given class and provide better insight into the relationship between the concepts and the model's output.

Given a training image, the concept embeddings of class $k$ are concatenated and passed through a sigmoid activated dense layer (parameterized by $\theta^R_k$) with a single output to learn the probability of classification for class $k$ as estimated by the pre-trained model. $\theta^R_k$ can be divided into chunks representing the connection weights for the individual concepts, i.e. $\theta^R_k=[\theta^R_{1k}, \ldots, \theta^R_{Ck}]$.  Then, the concept relevance for concept $j$ with respect to the output for class $k$ is denoted as $r_{jk}={\theta^R_{jk}}^T\textbf{e}_{jk}$. Note that the concept relevance $r_{jk} \in (-\infty, \infty)$. As the argument to the sigmoid activation of the dense layer is $\sum_{j=1}^Cr_{jk}$, the impact of the concept relevance scores on the prediction probabilities can be easily understood. Positive (negative) concept relevance increases (decreases) the prediction probability, thus emphasizing (reducing) the importance of the concept for a particular prediction.

The parameters of the dense layer, $\theta^R_k$, are learned by minimizing the cross-entropy between the output of the sigmoid and the pre-trained model's prediction probability for class $k$. Specifically, for a mini-batch of training instances, the loss for learning the parameters is defined as
\begin{equation}
    \mathcal{L}_k^R=-\sum_{i=1}^B f_k(i) \log\left(\sigma\left(\sum_{j=1}^Cr_{jk}(i)\right)\right)
\end{equation}
where $f_k(i)$ and $r_{jk}(i)$ are the pre-trained models output for class $k$ and the concept relevance for concept $j$ with respect to class $k$ respectively for the training instance $i$ .

\subsection{Output Generator}
To increase the faithfulness of the explanations we loop back the concept embeddings into the pre-trained model. The concatenated concept embeddings are passed through a dense layer (parameterized by $\theta^O$) to approximate the output, $\textbf{z}$, of the first dense layer of the pre-trained model. An $L_{2}$ loss between the approximation, $\tilde{\textbf{z}}$, and $\textbf{z}$, defined as $\mathcal{L}^D = \| \textbf{z} - \tilde{\textbf{z}}\|^{2}_{2}$ is used to learn the approximation. If the approximation is accurate we should obtain the output $f(\textbf{z})$ ($\textbf{x}$ is replaced by $\textbf{z}$, a slight abuse of the notation), when
$\tilde{\textbf{z}}$ is passed through the dense layers of the pre-trained model. This requirement is enforced by minimizing the divergence between the pre-trained model's outputs for $\tilde{\textbf{z}}$ and $\textbf{z}$ defined as $\mathcal{L}^O = KL( f(\tilde{\textbf{z}})  \|  f(\textbf{z}) )$. The $L_{2}$ loss is required to faithfully mimic the behavior of the pre-trained model. On the other hand, leaving out the divergence loss can still lead to a good approximation $\tilde{z}$, but $f(\tilde{z})$ can be inconsistent with $f(z)$.

Overall, the MACE framework minimizes $\mathcal{L}=\sum_{k=1}^K(\mathcal{L}^E_k + \mathcal{L}^R_k) + \mathcal{L}^D + \mathcal{L}^O$ with respect to the parameters $\theta^M, \theta^E, \theta^R$, and $\theta^O$. Adam optimizer \cite{adam} is used to minimize the loss function.

\section{Experiments}
We validate the MACE framework on VGG \cite{simonyan2014deep}, and ResNet \cite{DBLP:journals/corr/HeZRS15} architectures trained on the Imagenet dataset \cite{DBLP:journals/corr/RussakovskyDSKSMHKKBBF14}. The networks are fine-tuned on AWA2 dataset \cite{AWA2,awa2_paper_xian_2018_zero} and the Places365 dataset \cite{places}. We discuss the results for the AWA2 dataset using the VGG model in the paper and present the results for the Places365 dataset and ResNet architecture in the supplementary material (sections S8 and S7 respectively). The AWA2 dataset consists of 37322 images of 50 animal classes. We select a subset of 10 classes with approximately 400 images per class for our experiments. The VGG network that is fine-tuned with this dataset is our pre-trained model. For every class, we aim to learn 10 concepts ($C=10$) with the concept embedding dimension to be 32 ($Q=32$). Examples of the concepts extracted for various classes are presented in Figure \ref{fig:outputIllustration}. The architectural details of the embedding generator, along with the choices for the hyper-parameters are presented in the supplementary material section S1. Additional examples of the visualization of the concepts for the AWA2 dataset are presented in sections S3 and S4 of the supplementary material. While we describe the results from three salient experiments here, investigations into the stability and robustness of the explanations as well as ablation studies are discussed in sections S5, S6, and S9 of the supplementary material respectively.

\subsection{Faithfulness of the Explanations}
Faithfulness, the ability to accurately explain the working of a pre-trained model, is one of the most important traits of a post-hoc explanation model. As suggested in prior literature, we measure faithfulness using the drop in the classification probability upon masking. We generate a binary mask by thresholding each of the concept maps. As the concepts are trained to be different from each other, masking out just one concept might not give a significant drop in the probability of the predicted class, as the model can still predict with the help of other concepts. For example, as seen in Figure \ref{fig:ear_masked} masking the ear concept does not alter the classification probabilities significantly as the model compensates for it through the other concepts. Thus, we take the union of all binary masks. The region in the generated mask is removed from the image and the resulting drop in the probability of the predicted class is measured.  We also compare the drop in the prediction probability when the mask is synthesized using other saliency-map generating approaches as well as a random process (activation maps combined randomly). The results of this experiment are presented in Figure \ref{fig:faithfulness}. The highest drop in the probability is observed for the mask synthesized by the MACE framework for every threshold.
\begin{figure}[htb]
    \centering
\begin{subfigure}{0.495\textwidth}
  \includegraphics[width=\linewidth]{faith.png}
  \caption{}
  \label{fig:ear_masked}
\end{subfigure}\hfil
\begin{subfigure}{0.495\textwidth}
  \includegraphics[width=\linewidth]{faithfulness.png}
  \caption{}
  \label{fig:faithfulness}
\end{subfigure}\hfil
\caption{[Best viewed in color] Faithfulness of the explanations extracted by MACE (a) No change in the prediction distribution after masking a concept due to compensation by the model using other concepts, (b) Drop in the  probability of classification across different threshold values for generating the binary mask.}
\label{fig:faithfullness_full}
\end{figure}
\subsection{Explaining all the outputs for a test image}
An image classification network outputs a probability distribution over the class labels.  An important aspect of any explanation model is to explain all the non-zero probabilities in the output of the classifier. In our approach, this is explained by looking at the concepts generated for every class, and the corresponding concept relevances. We specifically look at the concepts that have positive concept relevance for any class. Figure \ref{fig:wrongexp} shows the results for a few images. For example, the first two rows present the explanations generated by different models for why the classifier has non-zero probabilities for the German Shepherd and Wolf classes when the correctly predicted class is Fox. The original image in the first column of \ref{fig:wrongexp} is a fox image, the visualization of explanation given by MACE (second column) for German Shepherd class and wolf class are meaningful and highlight smaller regions in the fox such as ears and legs, while other approaches give the same explanation for both the classes. We can also observe from the figure that MACE consistently gives different meaningful explanations for different classes, while other approaches give the same or no explanation.

\begin{figure}[htb]
    \centering
\begin{subfigure}{0.70\textwidth}
\centering
  \includegraphics[width=\linewidth]{wrong_class_imgs.JPG}
  \caption{}
  \label{fig:wrongexp}
\end{subfigure}\hfil
\begin{subfigure}{0.29\textwidth}
\centering
  \includegraphics[width=0.95\linewidth, keepaspectratio]{pvp.png}
  \caption{}
  \label{fig:pvp}
  \includegraphics[width=0.95\linewidth, keepaspectratio]{bargraph.png}
  \caption{}
  \label{fig:bargraph}
\end{subfigure}\hfil
\caption{[Best viewed in color] (a) contains explanations of different approaches for incorrect class. Each row corresponds to the explanation for some output probability for a wrong class. The wrong class is written on the left-most side in each row. The image shown for MACE approach is the visualization of the concept with the highest concept relevance for that class, and the concept relevance is written below the visualization; (b) Percentage of concepts with positive concept relevance less than the average score using only the images for different classes (See text for more details); (c) Average relevances of all the concepts for the true class and for all the classes except the true class.}
\label{fig:faithfullness_full}
\end{figure}

We further analyze the behavior of the concept relevance scores for different outputs of the pre-trained model. We first compute the average concept relevance for a concept embedding $e_{jk}$ using all the images belonging to class $k$. Over the test set, we compute the percentage of concepts with positive concept relevance less than the average score using only the images for which class $k$ is the second-highest ranked class according to the pre-trained model's predictions. We compute the average percentage across all classes for a particular rank. The result is summarized in Figure \ref{fig:pvp}. We can observe from the figure that the percentage of concepts with positive relevance score less than the average increases as the prediction ranking decreases. This is intuitive as concept relevance for the concept of an incorrect class should decrease as we go down the class ranking.

Figure \ref{fig:bargraph} shows the average concept relevances for all the concepts where the average is split between relevances of concepts belonging to the true class of a test image (AVG True) and relevances of concepts belonging to any other class except the true class (AVG Others). It can be observed that AVG Others (green bars) is significantly lower than AVG True (blue bars) for all the concepts. This further strengthens the credibility of concept relevances as concept relevance should be higher only if the concept actually belongs to the true class of the image than any other class.

\subsection{Human Evaluation of Explanations}
We evaluate the quality of our explanations generated for the VGG16 network fine-tuned on 10 classes of the AWA2 dataset. We present to every user the saliency maps generated by MACE, GradCAM \cite{gradCAM}, GradCAM++ \cite{grad_cam_plus_plus}, Excitation Backpropagation \cite{DBLP:journals/corr/Zhang0BSS16} and VisCNN \cite{vis_cnn} for a random set of 10 images. In addition, we randomly select 2 images from this set for repetition and create a final questionnaire of 12 examples and the corresponding explanations. The subjects were asked to choose the approach whose explanation helped them better understand the classifier's prediction. The responses in which the answers to the repeated questions mismatched were removed to maintain the consistency in the responses. We received 41 consistent responses, resulting in a total of 410 votes. The results have been summarised in Table \ref{HumanExptable}. We observe that explanations from MACE are the most preferred choice amongst the participants. Further, approaches like MACE and Excitation Backpropagation are preferred over gradient-based approaches like VisCNN, GradCAM and GradCAM++.
\begin{table}
\centering
\caption{Preferences of the Human Subjects for Explaining the Classifier's Predictions.}
 \begin{tabular}{||c | c||}
 \hline
 Approach & Vote Percentage \\ [0.5ex]
 \hline
 \hline
 MACE (ours) & \textbf{48.29\%} \\
 \hline
 Excitation Backpropagation \cite{DBLP:journals/corr/Zhang0BSS16} & 40.73\% \\
 \hline
 GradCAM++ \cite{grad_cam_plus_plus} & 9.51\% \\
 \hline
 GradCAM \cite{gradCAM} & 1.21\% \\
 \hline
 VisCNN \cite{vis_cnn} & 0.24\% \\
 \hline
\end{tabular}
\label{HumanExptable}
\end{table}

\section{Summary}
In this work, we define a new form of reasoning for explaining the outputs of an image classification network using multiple concepts/parts of an object. We present the MACE framework, for extracting and visualizing these multiple concepts. We also propose a mechanism for estimating the relevance of a concept towards the output of the model. We perform extensive experiments on the MACE framework using VGG16 and ResNet50 architectures for animal and places classification tasks. Our results confirm the faithfulness of the explanations as well as their human interpretability.

\newpage
\bibliographystyle{plainnat}

\section{Experimental Details}

Our implementation can be accessed at \url{https://github.com/mace19/MACE}. In this section, we discuss our experimental details for various architectures and datasets. For each dataset, we select 10 classes and for every class, we aim to learn 10 concepts with concept embedding dimension equal to 32.

\subsection{VGG16 on AWA2}
We train our MACE framework on VGG16 \cite{simonyan2014deep} architecture using a subset of AWA2 dataset \cite{AWA2,awa2_paper_xian_2018_zero}, having 10 classes with approximately 400 images per class. The learning rate is set to $10^{-4}$ and the model is trained for 64 epochs using an ADAM optimizer \cite{adam}. The accuracy of the model was 92.7\%

\subsection{VGG16 on Places365}
We train our MACE framework on VGG16 \cite{simonyan2014deep} architecture using a subset of Places365  \cite{places}, having 10 classes with approximately 5000 images per class. The learning rate is set to $5*10^{-4}$ and the model is trained for 32 epochs using an ADAM optimizer \cite{adam}. The accuracy of the model was 93.1\%

\subsection{ResNet-50 on AWA2}
We train our MACE framework on ResNet-50\cite{DBLP:journals/corr/HeZRS15} architecture using a subset of AWA2 dataset \cite{AWA2,awa2_paper_xian_2018_zero}, having 10 classes with approximately 400 images per class. The learning rate is set to $10^{-3}$ and the model is trained for 128 epochs using an ADAM optimizer \cite{adam}. The accuracy of the model was 96.5\%

\section{Pruning}

The set of concepts generated by following the proposed methodology generate some concepts that are not meaningful. We prune these concepts so that only the meaningful concepts remain. We use the following strategy for pruning the concepts:

\begin{itemize}
\item We fetch the top $T$ images in terms of concept relevance, and if more than $S$ of those images don't belong to the same class as the concept, then we prune the concept. In our experiments, we set $T = 10$ and $S = 5$.
\item If a large part of test dataset has positive concept relevance for a concept, we prune that concept. In our experiments with AWA2 dataset, we pruned the concept if it had positive concept relevance for more than 50\% of the test images.
\item If the concept masks in almost the entire image, we prune the concept. In our case, we pruned the concept if on an average it masked in 95\% of the image.
\item If a concept has negative concept relevance for most of the images that belong to the same class as the concept, then we prune the concept. In our experiments we pruned the concept if less than 5\% of the images belonging to the same class had positive concept relevance.
\end{itemize}

After pruning the concepts, we fine-tune the MACE model. We use the fine-tuned model to perform all the experiments. Visualizations of some of the pruned concepts are shown in Figure ~\ref{fig:pruned_concepts_visualization}

\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/0_9/0_9_7_m_.png}
  \label{fig:ear}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/2_6/2_6_8_m_.png}
  \label{fig:ear}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_1_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_0/4_0_8_m_.png}
  \label{fig:ear}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_1/4_1_9_m_.png}
  \label{fig:ear}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/pruned_concepts/4_2/4_2_7_m_.png}
  \label{fig:ear}
\end{subfigure}
\caption{[Best viewed in color] Some of the pruned concepts. Each row contains prototypical images of a pruned concept.}
\label{fig:pruned_concepts_visualization}
\end{figure}

\newpage
\section{Concept visualization and relevance for true class prediction}
Figure \ref{fig:first_5} shows the visualizations of the learned concepts and their corresponding relevance values for a few images. We observe that for most images the background concept had very high relevance thus suggesting that it plays an important contribution in the model's prediction. We also observe some repetitive pattern in the visualizations of the learned concepts. Such repetitions have been removed from this figure.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{images/true_classes/true_class.png}
    \caption{[Best viewed in color] Concepts with relevances corresponding to the predicted class}
    \label{fig:first_5}
\end{figure}

\newpage
\section{Concept visualization and relevance for other class prediction}
One of the major contributions of our method is that we can provide rich explanations for why a non-zero prediction probability is assigned to the class other than that of the predicted class. Figures ~\ref{fig:fox_lion_face}, ~\ref{fig:fox_dog_ear}, ~\ref{fig:horse_dog_color}, ~\ref{fig:zebra_tiger},  ~\ref{fig:leopard_wolf_body}, ~\ref{fig:zebra_background}, ~\ref{fig:tiger_background}, ~\ref{fig:tiger_wolf_face} show few such explanations.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{other_classes/1.JPG}
    \caption{[Best viewed in color] The model is looking at the face region in the fox image in this concept. The visualizations for lion for this concept is also face.}
    \label{fig:fox_lion_face}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/2.JPG}
    \caption{[Best viewed in color] The model is looking at the ear region in the fox image in this concept. The visualizations for dog for this concept is the ear and eye region. Since eyes are not clearly visible in the fox image, the model is looking at just the ears in this concept.}
    \label{fig:fox_dog_ear}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/3.JPG}
    \caption{[Best viewed in color] The bulges on back and front side of horse make the region highlighted in the explanation for german shepherd look like ears (based on color). Since the concept itself is also of ear and eye region, this concept contributes towards german shepherd probability in this image.}
    \label{fig:horse_dog_color}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/4.JPG}
    \caption{[Best viewed in color] In the concept in the first row, the model is looking at the background. Background of Zebra image and that of the images for the actual concept is quite similar. In the concept in second row, the model is looking at the stripes region in the zebra image, and the concept is also the stripes on tiger.}
    \label{fig:zebra_tiger}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/5.JPG}
    \caption{[Best viewed in color] The model is looking at the body of leopard in the explanation for wolf class. The concept itself also represents the body of wolf.}
    \label{fig:leopard_wolf_body}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/6.JPG}
    \caption{[Best viewed in color] The model is looking at the background. Background of Zebra image and that of the images for the actual concept is quite similar}
    \label{fig:zebra_background}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/7.JPG}
    \caption{[Best viewed in color] The model is looking at the background. Background of Tiger image and that of the images for the actual concept is quite similar}
    \label{fig:tiger_background}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{other_classes/8.JPG}
    \caption{[Best viewed in color] The model is looking at the face of tiger in the explanation for wolf class. The concept itself also represents the face region of wolf}
    \label{fig:tiger_wolf_face}
\end{figure}

\section{Stability}
Stability of an interpretation refers to visual comparison of the learned concepts for images of the same class with varying orientation, size or position. The explanations of such images should be similar as well i.e., the learned concepts should be consistent. Given the saliency maps of a few such images, we should be able to understand the underlying concept easily. Figures \ref{fig:outputIllustration1} and \ref{fig:outputIllustration2} show that our learned concepts have very high stability. We also demonstrate that the concept embeddings for a particular concept for images with varying orientation, size or position are very close to each other. We take a small set of 10 images from the fox class and randomly choose 5 concepts of this class. For each concept we calculate the pairwise Euclidean distance between the concept embedding of the images. The results are shown in Figure \ref{fig:eucl_dis} in the form of a box diagonal matrix. Let $c_i$ denote the concept and $f_i$ denote the image then the axis of the matrix are defined as $[(c_1,f_1),..(c_1,f_{10})), (c_2,f_1),..(c_2,f_{10}),..(c_5,f_1)..(c_5,f_{10})]$. As it can be seen that the distances along the diagonal is lesser compared to the non-diagonal distances. This shows that the embeddings of similar (dissimilar) concepts are closer (farther).

\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_0_9_m_.png}
  \caption{Fox - Ear}
  \label{fig:ear_fox}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/0_7_9_m_.png}
  \caption{Fox - Background}
  \label{fig:background_fox}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_7_8_m_.png}
  \caption{German Shepherd - Eyes}
  \label{fig:eyes_german_shepherd}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/1_9_9_m_.png}
  \caption{German Shepherd - Face}
  \label{fig:face_german_shepherd}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_3_8_m_.png}
  \caption{Horse - Crest}
  \label{fig:crest_horse}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/2_8_9_m_.png}
  \caption{Horse - Legs}
  \label{fig:legs_horse}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_2_9_m_.png}
  \caption{Leopard - Texture}
  \label{fig:texture_leopard}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/3_6_9_m_.png}
  \caption{Leopard - Body}
  \label{fig:body_leopard}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_4_8_m_.png}
  \caption{Lion - Body and Background}
  \label{fig:body_background_lion}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/4_8_9_m_.png}
  \caption{Lion - Face}
  \label{fig:face_lion}
\end{subfigure}
\caption{[Best viewed in color] Visualization of concepts across multiple images of the same class}
\label{fig:outputIllustration1}
\end{figure}

\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_3_7_m_.png}
  \caption{Tiger - Body and Background }
  \label{fig:body_background_tiger}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/7_6_9_m_.png}
  \caption{Tiger - Face \& Upper Body}
  \label{fig:face_upper_body_tiger}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_7_7_m_.png}
  \caption{Wolf - Legs }
  \label{fig:legs_wolf}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_1_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/8_9_9_m_.png}
  \caption{Wolf - Muzzle}
  \label{fig:muzzle_wolf}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_2_9_m_.png}
  \caption{Zebra - Background}
  \label{fig:background_zebra}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/stab/9_7_7_m_.png}
  \caption{Zebra - Stripes}
  \label{fig:stripes_zebra}
\end{subfigure}
\caption{[Best viewed in color] Visualization of concepts across multiple images of the same class}
\label{fig:outputIllustration2}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.45\textwidth]{images/draft4.png}

    \caption{[Best viewed in color] Pairwise Euclidean distance between concept embeddings}
    \label{fig:eucl_dis}
\end{figure}

\section{Comparison of robustness}

Robustness is an important desideratum of an explanation. The explanation generated by any interpretability method should be robust to local perturbations in the input image. Figure \ref{fig:robustness_exp} shows that this is not the case for popular interpretability methods; even adding minimal noise to the input introduces visible changes in the explanations. We formally quantify this parameter of evaluation as the intersection over union between the explanations of the image and its perturbed variation:
\begin{equation}
R(x_{i}) = IoU( f_{expl}(x_{i}) , f_{expl}(\tilde{x}_{i}) )
\end{equation}
Here $x_i$ is an input image and $\tilde{x}_i$ is the perturbed image. $f_{expl}$ refers to the explanation function whose output is the saliency map for the interpretation method. The saliency maps are thresholded to create a binary map before calculating the IoU. We perform robustness experiment on various parameters including variations in brightness, contrast, random noise and rotation. For each parameter we define a range of values that describes the intensity of perturbation, eg. standard deviation of the noise distribution to be added in an image, delta value to increase the brightness/contrast in the image, angle of rotation etc. We define a wide range of threshold values in the range $[0.3-0.7]$ to generate the binary maps and compare their robustness for three approaches including MACE, GradCAM \cite{gradCAM} and GradCAM++ \cite{grad_cam_plus_plus}.

\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
    \includegraphics[width=0.24\linewidth]{images/rob/2_4.png}%
    \includegraphics[width=0.24\linewidth]{images/rob/3_4.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/4_4.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/5_4.png}
\caption{Threshold = 0.4}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.24\linewidth]{images/rob/2_5.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/3_5.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/4_5.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/5_5.png}
  \caption{Threshold = 0.5}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.24\linewidth]{images/rob/2_6.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/3_6.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/4_6.png}%
  \includegraphics[width=0.24\linewidth]{images/rob/5_6.png}
  \caption{Threshold = 0.6}
\end{subfigure}

\includegraphics[width=0.5\textwidth]{images/rob/leg.JPG}
\caption{[Best viewed in color] Intersection Over Union for thresholded saliency maps of original and perturbed image }
\label{fig:robustness_exp}
\end{figure}

\section{Places Experiments}
The visualizations for the VGG model, trained on Places365 are shown in Figure \ref{fig:places_365_concepts}. We observed that most of the concepts were pruned and we had just 2-3 useful concepts per class. One of the reasons could be that the classes in the dataset were different from each other and didn't require a large set of concepts to make the prediction.

\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_0/0_0_7_m_.png}
  \label{fig:airfield_airplane}
  \caption{Airfield - Airplane concept}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/0_2/0_2_9_m_.png}
  \label{fig:airfield_background}
  \caption{Airfield - Background}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/1_1/1_1_7_m_.png}
  \label{fig:airplane_cabin_seats}
  \caption{Airplane Cabin - seats in the airplane cabin}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_3_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_5_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/2_0/2_0_6_m_.png}
  \label{fig:airplane_cabin_top_lights}
  \caption{Airplane Cabin - Lights on top in airplane cabin}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_6_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_6/4_6_8_m_.png}
  \label{fig:alley_light_region}
  \caption{Alley - Region with light coming from the end of alley}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/4_2/4_2_9_m_.png}
  \label{fig:alley_walls}
  \caption{Alley - Walls of houses}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_1_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/5_3/5_3_9_m_.png}
  \label{fig:amphitheater_curvature_steps}
  \caption{Amphitheater - The curvature and steps in amphitheater}
\end{subfigure}
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_7_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_8_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/places/6_0/6_0_9_m_.png}
  \label{fig:amusement_arcade_games}
  \caption{Amusement Arcade - Games region in amusement arcade}
\end{subfigure}
\caption{[Best viewed in color] Concepts for Places365 Dataset.}
\label{fig:places_365_concepts}
\end{figure}

\section{ResNet Experiments}
The visualizations for the ResNet model, trained on AWA2 are shown in Figure \ref{fig:resnet_outputs}. We observed more head and body concepts in the case of the ResNet model, as compared to the VGG model. A possible reason for this could be that ResNet is a much deeper network than the VGG, and hence the features present in the activation maps of the last convolution layer majorly contain high-level body concepts.
\begin{figure}[H]
    \centering
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_1_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_4_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_7_m_.png}%
   \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_head/0_3_8_m_.png}%

  \caption{Fox - Head}
  \label{fig:fox_head}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_2_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_7_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_0_8_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_3_7_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/zebra_legs/9_3_2_m_.png}%

  \caption{Zebra-Legs}
  \label{fig:zebra_legs}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_5_2_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_5_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_5_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_5_8_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_5_9_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_6_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_6_1_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/lion_face/4_6_3_m_.png}%

  \caption{Lion-Head}
  \label{fig:lion_head}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_7_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_8_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/horse_body/2_9_9_m_.png}%

  \caption{Horse-Body}
  \label{fig:horse_body}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_0_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_5_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_7_m_.png}%
   \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_8_m_.png}%
   \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/wolf_head/8_3_9_m_.png}%
  \caption{Wolf-Head}
  \label{fig:wolf_head}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_2_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_3_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_4_m_.png}%
    \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_6_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_7_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_8_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_9_m_.png}%
  \hfill
  \includegraphics[width=0.12\linewidth]{images/resnet/tiger_body/7_1_5_m_.png}%
  \caption{Tiger-Body}
  \label{fig:tiger_body}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_0_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_1_m_.png}%
    \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_2_m_.png}%
    \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_6_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_4_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_7_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_8_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/fox_back/0_6_9_m_.png}%
  \caption{Fox-Back}
  \label{fig:fox_back}
\end{subfigure}
\\
\begin{subfigure}{\textwidth}
  \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_0_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_1_m_.png}%
    \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_2_m_.png}%
    \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_3_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_4_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_5_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_7_m_.png}%
  \hfill
   \includegraphics[width=0.12\linewidth]{images/resnet/cat_face/5_0_9_m_.png}%
  \caption{Cat-Face}
  \label{fig:cat_face}
\end{subfigure}
\caption{ResNet Outputs}
\label{fig:resnet_outputs}
\end{figure}

\newpage
\section{Ablation Study}

In order to justify the need for both the $\mathcal{L}^O$ and $\mathcal{L}^D$ losses, we compared the faithfulness of the approach with and without these losses. According to our hypothesis, we require the $\mathcal{L}^O$ loss to help us recreate the output of the first dense layer of a classifier, so as to keep our concept embeddings faithful to the classifier. We require $\mathcal{L}^D$ Loss to avoid possibility of inconsistency with the final output of the classifier.

We train two models, one without using the $\mathcal{L}^O$ loss, while other without the $\mathcal{L}^D$ loss. We train both the models for 50 epochs, setting the learning rate to be $10^{-4}$.

In Figure \ref{fig:ablation}, we see that there is a significant difference in drop in the true class probability when either of the two losses are reduced, showing both these losses are necessary for the faithfulness of our MACE unit.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{abalation.png}
    \caption{[Best viewed in color] Effect of $\mathcal{L}^O$ and $\mathcal{L}^D$ losses on model faithfulness}
    \label{fig:ablation}
\end{figure}

\newpage
\bibliographystyle{plainnat}

redpajama_set_name: RedPajamaArXiv


text:
Guinness Certifies World's Heaviest Strawberry
The Ilan berry clocked in at 289 grams (about 10 ounces), whereas an average strawberry weighs 15 grams.
By Abigail Klein Leichman February 22, 2022
The record-breaking strawberry measured 7 inches long, 1.57 inches thick and 13.3 inches in circumference. (Shalev Ariel)
On February 12, an Israeli strawberry was certified by the Guinness World Records as the world's heaviest.
Toot BaSadeh (Strawberry in the Field) farm, in centrally located Kadima-Zoran, grew the berry that weighed a bit more than 289 grams (about 10 ounces), or 299 with its stem.
To put that in perspective, farmer Tzahi Ariel first weighed an iPhone, which clocked in at 194 grams.
A regular strawberry weighs just 15 or 16 grams. The previous record-holder, grown in 2015 by Koji Nakao in Japan, weighed 250 grams.
The massive berry measured 18 centimeters (7 inches) long, 4 cm (1.57 inches) thick and 34 cm (13.3 inches) in circumference.
"I am a second-generation strawberry farmer," Ariel tells ISRAEL21c. "Since my three brothers and I were children, we grew strawberries with our father."
About 10 varieties of strawberries are raised on Toot BaSadeh's 100 dunams, or 24.7 acres. The record-breaking Ilan variety was first planted there about three or four years ago, and like the others is sold in the farm's own market and across Israel.
Israeli strawberry farmers Yedid Ariel, left, and Tzahi Ariel with the record-breaking berry in February 2022. (Shalev Ariel)
"The news has spread all over the world. I am very proud to put Israel and Kamida-Zoran on the map," Ariel says.
"We are getting calls from China, Australia, Dubai — everywhere. It's a big honor for us."
Named for kids
Nir Dai, the Volcani Center plant scientist who bred the Ilan variety grown on Toot BaSadeh, was there for the official weighing.
This strawberry expert said it was indeed the biggest strawberry he'd ever seen.
Nir Dai of the Volcani Center displaying Ilan strawberries with son Yonatan. (Courtesy of Nir Dai)
Although Israel has only about 1,000 acres of strawberries — all grown in covered hothouses at various farms — the government's renewed breeding program at the Volcani Center has introduced about 10 commercial varieties since 2006.
Like Ilan, almost all new Volcani cultivars are named in honor of (and a few in memory of) children: Tammuz, Barak, Yasmin, Gili, Lavi, Tamir, Ella and Rocky, named for Dai's son Sela, which means "rock" in Hebrew. The Carmel Agrexco company bred strawberries named Orly, Yuval and Dandy.
Tamir hanging strawberries are grown without soil. (Courtesy of Nir Dai/Volcani Center)
"Matan and Rocky are very good, and now they dominate the fields," Dai tells ISRAEL21c.
One of the Volcani Center's original goals in breeding new varieties was lengthening the growing season. "In the 1970s, you could find strawberries in the market only from late January to mid-February," Dai says. Now Israeli strawberries are available from November to May.
Rocky strawberries, named after the breeder's son. (Courtesy of Nir Dai/Volcani Center)
The focus today is enhancing shelf life, flavor, aroma and firmness; berries that are too soft cannot be shipped long distances.
Shape matters, too – for example, Lavi is wide and conical while Hadar is oval, and Tammuz is heart-shaped. Color is another consideration depending on the market. The British prefer bright red strawberries, while Americans like darker ones.
Lavi strawberries weigh 28-30g and have a bright red color and sweet taste. (Courtesy of Nir Dai/Volcani Center)
However, Dai says, most Israeli strawberries stay on the domestic market because demand is high and there isn't enough land to increase capacity significantly.
The record-breaking Ilan, planted in October, is very vigorous and flowers rapidly. In the very early stage, several fruits fuse together into one flower, Dai explains.
"This variety always has huge berries, 50 to 60 grams. And the weather very much affects the size. Cold temperatures make the berry big and heavy because from flower to fruit takes more than 45 days, compared to 25 days in warmer weather."
Dai says the Volcani Center experimented with growing strawberries in the Golan Heights in July, but their average weight was just 12 to 15 grams.
He has high hopes for Ilan, which is easy to pick and boasts exceptional flavor and shelf life.
"It's very crunchy and sweet and hardy enough to grow outdoors. I hope it will be commercialized in southern California or even in South Africa," Dai says.
Produced in association with ISRAEL21c.
VIDEO: Zoo Gets Rare Chameleons To Breed After They Were Smuggled From Africa In Socks»Next Post
VIDEO: Cops Arrest 2 Over Irreparable Damage At Unique Crystal Cave In Spain

redpajama_set_name: RedPajamaCommonCrawl


Paramore has definitely moved into the 80's influenced alt-pop space that everyone seems to so desperately want to occupy. While I prefer the angst that was brought before the lineup shakeup in their first three albums, I do like the slightly more complex percussion re-administered by the return of Zac Farro.
A rock band making a dance record....I think Paramore's Hayley has been hanging out with too many producers trying to find that perfect mix of guitar driven pop music. You want to applaud the ambition to keep trying to do something new but this has been done before by bleachereds and the like. All sorts of aggro have been lost on this band.
Paramore is still making records, huh? Alright. Well, it sounds like someone fell asleep watching Bruuno Mars videos, woke up in a puddle of pink goo, and then heard Duran Duran on their Spotify "Discover Weekly" playlist, and thought to themselves, "hey, I can do that!"


Non-residents, who have one or more children born in Panama who are at least 5 years old, are eligible to apply for Panamanian permanent residency. This process requires the services of a Panamanian lawyer.
If approved, the applicant will receive a two-year temporary residency permit. Once the two-year period is over, the resident will be able to request permanent residence.
Contact Us to learn more or to start with your Parents of a Panamanian Children Visa process.


text:
Preparing Darlington for Winter
We get a fair amount of snow at Darlington during most winters. Although only two hours north of Manhattan by car, the climate in our village in the Hudson River Valley is appreciably-colder than in the City. As the winter season takes hold more often than not we'll find ourselves blanketed with snow when it's raining in the more-temperate City.
To prepare for the snowy season a lot of planning and industry is required. While our property is not particularly large, sitting on only two acres in a village setting, there is much to do to get it ready for winter. Once the weather turns cool the screened porch is emptied of its furniture, which is stored in our basement, and its sun-blinds are taken down and put away in the barn. The most critical task is to have the property raked of the leaves that fall from our many trees, mulch the beds, and amend the lawns, which takes weeks of work to complete. While that's going on we empty and store the large clay pots in which we grow herbs and decorative plants during the summer, and clean and store away the gas grill. We also empty and then re-plant the cast iron urns with evergreens for "winter interest", and store the wooden garden furniture in the barns. Water in the outbuildings and spigots is turned off and the pipes are drained. Gutters are cleared of debris. Reflectors are placed along the perimeters of the drives, and vulnerable shrubs along the village road are protected from salt and spray with burlap-wrapped snow-fencing. Firewood is procured and stacked away from the weather, the wood storage box is brought up to the house from the barn and stocked, and the wood baskets within the house are brought out of storage and filled, ready for the first fires to be lit in any of our six working fireplaces.
I am the one in primary charge of engaging and over-seeing the necessary support to make this happen at Darlington. Managing the logistics and timing is not without challenges, particularly since we live in the City during the week. Given the relatively short window that exists between when the last leaves have fallen and the snow begins, this can be somewhat nerve-wracking. But it is highly satisfying when I succeed in buttoning up our property before the first snowflakes flutter down, usually to remain in place until the spring thaw.
A lot of manpower is required to prepare the property for the winter. Boy is responsible for emptying and planting the urns (more on that in another post). Other than that we depend on others to do the lion's share of the work. The bulk of the leaf clean-up is done by the locally-based county lawn service company that takes care of our lawns during the summer. They come four or five times during leaf fall to rake and remove leaves from our property. We are happy for them to take the leaves away since we know they will compost them as opposed to sending them to land-fill. It is an impressive sight to behold when their trucks arrive and dis-engorge half a dozen workers armed with rakes and blowers, and to see them in action. They also put up the snow-fencing and burlap along the village road.
But there's more. Once the service has completed the bulk clean-up of the leaves, we then have our handyman, named "Rich", come through the property and do a more-detailed leaf removal, plus most of the other projects on our checklist. He is usually accompanied by his teenage son, until recently known as "Junior" but more recently as "Richard", and often another worker when the job demands it. Rich works for us on most weekends, usually Saturdays, and is someone whom we rely on for much of the work done maintaining our property. We first engaged him shortly after buying Darlington to plow our drives, which he still does, and over time he has taken on more and more responsibilities for us. He's a great guy, we like him a lot, and he is someone that we feel blessed to have in our lives.
Rich and his son will spend at least two-to-three Saturdays doing detailed leaf removal after the lawn service has done their bulk removal, and when he's done the property looks almost as manicured as a golf club, or as our friend Warner says: "coiffed". They also mulch the beds, store the garden furniture, clean out the gutters, place the reflectors along the drives, and take care of the firewood. We have our plumber shut off the water to the outbuildings and blow out the pipes so they won't burst. He's best-equipped to do this, since he was the one who installed the exterior plumbing in the first place.
This year we were able to get all of the preparations completed ahead of the first snowfall. But we really timed it down to the wire: Rich and his crew of Richard and their friend Tony finished raking up the last bit of leafage just as the first snow flakes started to come down last weekend. Since then we've had another snowfall, and it looks as if the ground may well be covered with snow until the spring.
Maintaining a property like Darlington well requires attention, labor, good-timing, organization, and resources, too. I enjoy making it happen and appreciate that it wouldn't be possible without the support of those who do the hard work and take pride in a job well-done. I consider our stewardship of Darlington to be much more than a responsibility: it's a privilege.
Labels: Darlington, gardening
SKY PRINCESS September 29, 2010 at 10:15 PM
I appreciate autumn leaves so much more because I do not have to rake them! I never realized so much had to be done in the colder states for winter. Your photo of the snow and the walkway is gorgeous.
We Wish You a Happy 2010
It Takes An Army To Throw A Party
Darlington As Decorated As It's Going to Get, Part II
Darlington As Decorated As It's Going to Get, Part I
The Decorating Progresses...
My Darling Clementine(s)
A Christmas Gift for Pompey
A Christmas Greeting From Long Ago...
The Decorating Begins...
Boy Makes a Wreath
Guest-Posting on Emily Evans Eerdmans
My Slim Keith Story
A Darlington Dinner Party
Lunch at the Russian Tea Room
Darlington House
The Joys of Old-Fashioned Calendars
Reggie's Five Favorites: The Series
Our Black Friday
Welcome to Reggie Darling


text:
Unpicking the Netflix quandary
By Chris Curtis2018-05-30T06:00:00+01:00
SVoD giant is conundrum for producers, broadcasters and analysts
When Netflix's market cap eased past the stock market's valuation of Disney last week (the two giants are neck and neck as I write), it felt both seismic and silly. The former has justifiably become the symbol of the peak TV era, and shorthand for the challenges facing the industry in terms of changing viewing habits and the power of a truly global player when so many British players feel just that – very British.
A very senior exec at a heritage UK broadcaster recently told me that Netflix doesn't have brand values in the same way that the PSBs do. He meant in terms of commitment to risk-taking, or innovation, or ordering shows that feel especially British – but I'm not sure he's right.
Netflix might not have a remit, but its brand identity is a commitment to creating content that is ostentatiously uber-premium. We haven't yet seen any of the SVoD operator's shows from Shonda Rhimes, Ryan Murphy or Barack and Michelle Obama, but when you strike deals with elite talent like that, you're definitely making a statement about your values.
Yet there's something baffling about the valuation of Netflix, in terms of its debt (significant) and profitability (small), compared with, say, Disney or Comcast. Maybe it's unfair to be critical of a business that is delivering incredibly high growth and that has attracted 125 million subscribers around the world, but its sky-high valuation could prove an albatross.
There was always an expected narrative that one of the established players would ultimately buy the upstart streamer – at its current price, that seems unlikely.
If the BBC can no longer match the financial might of the likes of Netflix, striving to be the best creative partner it can seems to be a smart response
In the meantime, BBC2 controller Patrick Holland has joined director general Tony Hall and his deputy Anne Bulford in asserting the BBC's cultural capital in the face of the Netflix revolution.
Ultimately, it's for producers to decide what type of deals to strike with the SVoD giant, and whether to take its cheque if they can't or don't want to carve out IP ownership. It's an old dilemma that goes back to dealing with lots of US cable operators and networks.
But Holland's point about creative partnerships is more nuanced and intriguing. There is a fallacy that Netflix doesn't offer notes, but producers do confirm its hands-off approach.
That makes interference a lot less likely – but rules out creative contribution as well. If the British commissioning model works properly, its distinctiveness should give rise to high-quality content that will benefit producer and broadcaster alike. If the BBC can no longer match the financial might of the likes of Netflix, striving to be the best creative partner it can seems to be a smart response.
Chris Curtis is the editor of Broadcast
Netflix to open London commissioning base
Diego Buñuel to head original docs at newly-created division
Patrick Holland: Breathing new life into BBC2
Controller aims to reflect the diversity of modern Britain and turn up the dial on challenging religious and ethical programming
BBC2 boss sounds FAANG alarm
Indies risk 'selling the golden goose' by handing IP to SVoD players, says Patrick Holland
More Broadcasters
Former Reef chief resurfaces with listings service
Richard Farmbrough launches Must TV
Morning wrap: Peaky Blinders; Celebrity Catchphrase; Ranganation
News round-up including the end of Steven Knight's big-budget BBC drama
Amazon snaps up eOne thriller
Cruel Summer to stream globally outside North America after Freeform premiere


text:
eTechShout
TECH2 years ago
Handpick the Best of Souq Camera and Accessories at Amazing Offers
Souq, the online e-commerce portal of the Arab world brings discounts on cameras and accessories with amazing offers. From memory card to super-sturdy tripods, everything is...
Get your Hands on the Bestselling Indian Classics at Amazing Offers on Amazon
To all the book lovers out there, here Amazon brings in the best news for you! Get your hands on the bestselling Indian Classics at exciting...
Why Are APIs a Key Factor in Digital Transformation?
Nowadays, most companies are considering a digital transformation of their businesses. This is happening mainly because they are realizing that digital transformation will be effective in...
Explore the Globe with iOS & Android AR Strategy Game 'Domination: Earth'
New Android and iOS augmented reality strategy game "Domination: Earth" has launched, operating in the real world around you – military style! Check in wherever you...
MSI Infinite A Gaming Desktop Is a Worthy Addition To the Home
MSI makes worlds best motherboard and supports the NVIDIA graphics. The company is very popular for a durable product for gaming which are powerful, durable...
EXPLAINER4 years ago
Here's Everything You Need To Know About WhatsApp Bot Duta
Heard about Whatsapp Bot Duta? WhatsApp is a well-known and popular messaging App to connect with your friends and family. Nowadays it became more popular because of...
10 Best TV Guide Apps for Android and iOS in 2021
13 Best Auto Mouse Mover Software for Windows in 2021 (FREE)
15 Best Cut and Paste Apps for Android & iOS (2021)
Who Has the Best Internet in the World?
11 Best Age Progression Apps for Android and iOS (2021)
Copyright © 2021 All Rights Reserved. All Logos & Trademark Belongs To Their Respective Owners


Many weekend mornings when we were little my grandpa would make us pancakes. But these were not just ordinary pancakes. He was a pancake artist, making them into different shapes. Mostly cars or animals or Mickey Mouse ears. I just about thought this was the coolest thing in the world when I was 6 and honestly, I still do.
I also have this vivid memory of my sister and I sitting at the table gobbling down our pancakes and here comes our grandpa with a plate for our grandma with pancakes shaped in the letters: I L-O-V-E Y-O-U. I'm pretty sure that he thought we weren't paying much attention to this at the time, but even if he thought we were that wouldn't have made a difference. My grandparents are perfect examples of the way partners should treat each other even after they've been together for 40, 50, and 60+ years. Like they are newly in love and they do little things each and every day to show that.
So I guess what I'm getting at here is pancakes feel like love to me. They bring back all kinds of warm and fuzzy feelings of love and it just so happens that they are one of my husband's favorite breakfasts. But I didn't want to just make pancakes. I found a recipe for "Sour Milk Griddlecakes" in my vintage cookbook and tweaked the recipe a bit to create the most fluffy, melt in you mouth griddlecakes around. I also made a homemade syrup which tastes exactly like cinnamon rolls. Seriously. Put away that store bought maple syrup and make this. It is divine.
In a large bowl whisk together the dark brown sugar, cake flour, salt, and baking soda. Make a well in the center of the dry mixture.
In a medium bowl, whisk the eggs until fluffy. Beat into the eggs the buttermilk, vanilla extract, and melted butter.
Pour the wet ingredients into the dry ingredients and gently mix until combined. The batter will be lumpy. Do not over mix to a smooth consistency.
When ready to fry the griddlecakes, blend into the batter the baking powder and let sit for 5 more minutes.
Fry the cakes on a lightly greased and very hot griddle. When bubbles appear on the surface of the cakes, flip them before the bubbles burst. This will give you light and fluffy cakes.
Combine butter, brown sugar, buttermilk, vanilla, cinnamon, and corn syrup in a medium sauce pan. Bring to a simmer. Once simmering, mix in baking soda and cook for 10 seconds. Remove from heat.
Make the syrup before the pancakes and let it cool slightly so it will thicken up a bit. This syrup will not get as thick as regular maple syrup.
this looks DELICIOUS (and i'm a fan of photographing pancakes as well)! i made some buckwheat pancakes this morning, but next time i'll have to try the cinnamon roll syrup!
i stumbled across this blog this morning under the food topic, and i'm glad i did!
Thank you! I wasn't too happy with the way these pics turned out but my husband was ready to eat so I didn't redo them. I'll have to check out your blog!
I think your pics look fabulous! I am still in bed and now I'm just wishing someone was in my kitchen making me those delicious pancakes and putting them on a tray and serving them to me right here!
Bettie, those look incredible – what a perfect breakfast! The syrup sounds delicious.
My mouth is watering already! I can't wait to make them because they look and sound delicious!
Great photos, great stories, great recipes. I will be waking up to these griddlecakes some time in the near future.
These look so yummy. Yet another one of your recipes to add to my "to try" list! Funny how you mention your grandpa was a pancake artist. The Hubby and I were discussing all the shapes we should try to turn pancakes into over the weekend. I think he was challenging me, becuase I said I don't know if i would do so well turning batter into shapes. I think I will have to prove him wrong next Saturday with this recipe!
Art has never been one of my fortes. I think my only creativity lies in creating dishes. I'm not very good at the decorating. You'll have to let me know how they turn out!
Ha! And they really do taste just as good as they look. I don't know how I worked my magic on that syrup. But I don't think I can ever go back to store bought now. SO GOOD!
My husband thanks you in advance. His mind shall be blown this weekend a la this recipe.
Ugh, I just LOVE your blog so much!! Your grandpa sounds like my uncle–he used to make us pancakes in our initials and I just absolutely adored that tradition! Now, if someone started making these pancakes with that cinnamon roll syrup, then I would love that even more 🙂 These look insanely fabulous. xo!
oh, you are dangerous. cinnamon roll syrup? I may just have to make this tomorrow morning…and my father also made fun pancake shapes for us as children. It's the simple memories that stick.
We had breakfast for dinner this evening & used your recipe. Delicious! This will now be our family's go-to recipe. The cinnamon roll syrup is divine!
they got the best deal when they actually are mistaken.
Oh, MAN! This stuff is good!! Thank you ma'am for dreaming up this recipe and sharing it! I'm glad I'm a big girl now, and it's up to me whether I eat pancakes WITH CINNAMON ROLL SYRUP every day. No kiddin'. For years now, I've been fairly indifferent to pancakes/waffles, but that all changed this morning. You are a genius! Now I can have the flavor equivalent of a fresh, homemade cinnamon roll with the ease of making pancakes any time I want! I'm going to bed right now, so I can get up in the morning and have more pancakes AND CINNAMON ROLL SYRUP!! I predict very sweet (and buttery-brown-sugar-cinnamony) dreams in the interim. . .
I'm sooooo happy you enjoyed these!
The recipe isn't showing up – any chance of fixing the link? I would love to make these for my family!
Thanks for bringing this to my attention. My site went through some updates recently and some of my really old recipes got deleted. I have updated the post to remedy the problem!
Yes, I see a couple of broken links, but the recipe is missing. The blog and all the photos are there but no recipe.
Hi Judy! Thanks for letting me know! My site went through some updates recently and some of my old recipes got deleted. I have fixed it now!
Thanks so much for fixing it! Can't wait to try these!