\section{Introduction}
\label{introduction}
\IEEEPARstart{T}{hough} convolutional neural networks (CNNs) have shown predominant performance in image classification tasks~\cite{simonyan2015very, he2016deep}, the vast demand on computation cost has prohibited them from being deployed on edge devices such as smartphones and embedded sensors. To address this, the researchers have developed several techniques for CNNs compression, such as network pruning~\cite{han2015learning,ding2019global}, parameter quantization~\cite{hubara2016binarized, liu2020bi}, tensor decomposition~\cite{peng2018extreme, hayashi2019exploring} and knowledge distillation~\cite{romero2014fitnets,hinton2015distilling}, \emph{etc}. Among them, channel pruning has attracted ever increasing attention for its easy combination with general hardware and  Basic Linear Algebra Subprograms (BLAS) libraries, which is thus the focus of this paper.

channel pruning removes the entire channels to generate a sub-network of the original CNN with less computation cost. Existing studies could roughly be categorized into three categories. The first group abides a three-step pruning pipeline including pre-training a dense model, selection of ``important'' filters and fine-tuning the sub-net. Typically, most works of this category focus on the second step by either figuring out a filter importance estimation, such as $\ell_1$-norm~\cite{li2017pruning}, geometric information~\cite{he2019filter} and activation sparsity~\cite{hu2016network}, or regarding channel pruning as an optimization problem~\cite{lin2020hrank, guo2020channel}. The second category implements channel pruning with additional sparsity constraints~\cite{liu2017learning, luo2020autopruner, ding2019centripetal, ding2020lossless}, after which, the pruned model can be available by removing zeroed channels or channels below a given threshold. The last group applies AutoML techniques to directly search the channel number of each layer under a given computation budget, thus achieve pruning in an automatic manner~\cite{he2018amc, yang2018netadapt, liu2019metapruning, liu2021joint}.

\begin{figure*}[h]
\begin{center}
\includegraphics[width=0.85\linewidth,height=0.2\pdfpageheight]{Visualization.jpg}
\end{center}
\vspace{-1.0em}
\caption{\label{visualization}Visualization of images from different categories (first row) along with the feature maps of the 5-th and 144-th channels (second and third row, respectively) in the conv-12 layer of VGG16-Net~\cite{simonyan2015very} trained on ImageNet.
As can be seen, with different scene, light, pose of images as input, head and textual information are always activated by these two channels respectively, despite that the ImageNet dataset does not contain such categories.
All feature maps are with their original activation value and size.
\vspace{-1.5em}
}
\end{figure*}

Despite the progress, existing methods build channel pruning by observing the CNN output, \emph{i.e.}, the final classification performance, while leaving the internal influence of a CNN model hardly touched. For example, Li \emph{et al}.~\cite{li2017pruning} removed filters with smaller $\ell_1$-norm, which can indeed be viewed as to minimize the output difference between the original model and the pruned model. To take a more in-depth analysis, the massive non-linear operations inside CNNs make them hardly understandable. Thus, existing methods~\cite{li2017pruning, ding2019approximated} choose to regard the CNNs as a black box and observe the final output for network pruning.
For instance, Ding \emph{et al}.~\cite{ding2019approximated} leveraged binary search to remove filters with the least accumulated errors calculated by the final output.
From this perspective, we term these methods ``Black-Box pruning'' in this paper.

Nevertheless, understanding the internal explanation of deep CNNs has attracted increasing attention~\cite{wu2017interpretable,yosinski2015understanding,zeiler2014visualizing,zhang2018interpretable,zhou2014object}, which also advances various vision tasks. For instance, Zeiler \emph{et al.}~\cite{zeiler2014visualizing} won the championship of the ILSVRC-2013 by adjusting architecture through visualization of internal feature maps. Inspired by this, we believe that exploring the internal logic in CNNs could be a promising prospect to guide channel pruning.

As exploited in~\cite{yosinski2015understanding}, the feature maps of each channel have the locality that a particular area in one feature map is activated. Inspired by this, we visualize the feature maps generated by VGG16-Net~\cite{simonyan2015very} trained on ImageNet to explore the local information in the internal layers of CNNs. As can be seen from Fig.\,\ref{visualization}, the 5-th channel at the 12-th convolutional layer always generates feature maps that contain head information while the 144-th channel attempts to activate textual information. Even though there is no explicitly labeled head or text, this CNN model automatically learns to extract partial information to make better decisions, which exactly meets human intuition when classifying an image. That is, head information extracted by the 5-th channel helps the network to identify animals, and textual information extracted by the 144-th channel contributes to classify categories with texts such as digital watches. However, some local features may not be beneficial to identifying all categories. For example, the 144-th channel always chooses to deactivate most of the pixels when processing images with no textual semantics like dogs and pandas (see the third and fifth columns in Fig.\,\ref{visualization}). Such local representation on the internal layers of a CNN shows that channels have varying contribution to different categories in image classification, which motivates us to rethink the importance criterion of channel pruning.

Instead of simply considering the CNN output after removing a channel as prior arts do, we target at finding each channel's contribution to identifying different kinds of images. It is intuitive that if feature maps activated by one channel can benefit most categories' classification, this channel is essential and should be preserved; otherwise, it can be safely removed.

To this end, we assign each channel a class-wise mask, the length of which is basically the same as the category number in the training set. Specifically, we utilize the ground-truth labels as auxiliary information in pruning the network. For each category of the input images, the corresponding mask is activated to multiply on the output feature map for model inference. If one channel generates feature maps that have a positive recognition effect on some categories, masks corresponding to these categories will receive large training loss gradients. By exerting a sparsity constraint that pulls the class-wise mask toward zero to counteract such gradients, these masks will maintain relatively large absolute values. On the contrary, if this channel contributes little to most categories, then the corresponding masks will be punished close to zero. Thus, after a few training epochs, each channel's importance score can be measured by the absolute sum of its class-wise mask, reflecting its overall contribution to identifying all categories. In this way, we can carry out the pruning in an explainable manner, for which we term our pruning as ``White-Box".

We further propose an iteratively global voting, which is performed using the above importance score to remove unimportant channels until the FLOPs of the pruned model meet the pre-given computation budget. It is worth mentioning that the layer-wise pruning rate can be decided in an automatic manner, which demonstrates the efficiency of White-Box compared with the previous works using hand-crafted designs~\cite{li2017pruning,lin2020hrank} or conducting time-consuming search~\cite{he2018amc,liu2019metapruning}. Lastly, a fine-tuning process is conducted to recover the performance of the pruned network.

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=0.8\linewidth,height=0.17\pdfpageheight]{framework.pdf}
\end{center}
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{0pt}
\vspace{-1.0em}
\caption{\label{framework}Framework of the proposed White-Box for class-wise mask training. For better representation, we choose one channel for description. Our White-Box will assign a class-wise mask for this channel with a dimension of category number $D$ in the dataset, which aims at measuring the contribution of this channel to different categories' identifying. With a specific category of the input image, we first soft the one-hot vector of the ground-truth label to activate the class-wise mask for stable training. Output feature map of this channel will then be multiplied by the class-wise mask, and be added in an element-wise manner to generate the final feature map for model inference. By doing so, the class-wise mask will receive gradient signals respecting to input images' category. Therefore, White-Box will be able to find this channel's contribution to recognizing different categories. (Best viewed with zooming in)
\vspace{-1.5em}
}
\end{figure*}

Our contributions are summarized as follows:
\begin{itemize}
\item Based on an in-depth analysis of CNNs interpretation, we propose a novel explainable importance criterion for channel pruning that we should preserve channels beneficial to identifying most categories. To our best knowledge, this is the first time that CNN interpretability theory is considered to guide channel pruning.
\item We carry out channel pruning in a white box by jointly training a class-wise mask along with the original network to find each channel's contribution for classifying different categories. Then a global voting and a fine-tuning are conducted to obtain the final pruned model.
\item Extensive experiments on CIFAR-10 and ILSVRC-2012 demonstrate the advantages of the proposed White-Box over several state-of-the-art advances in accelerating the CNNs for image classification.
\end{itemize}

\section{Related Work}

\textbf{Channel Pruning.}
Channel pruning targets at snipping away entire channels in convolution kernel to obtain a pruned model, which not only saves computation cost, but is also compatible with off-the-shelf hardware. As discussed in Sec.\,\ref{introduction}, previous channel pruning works can be approximately divided into three groups.

Starting from a pre-trained model, the first category designs various importance criteria to remove unimportant channels. For example, Li \emph{et al}.~\cite{li2017pruning} chose to prune filters with smaller $\ell_1$-norm. SASL~\cite{shi2020sasl} proposed to measure the importance of channels using both the prediction performance and computation consumption. He \emph{et al}.~\cite{he2017channel} considered the construction error of the next layer as an importance criterion and conducted pruning in a layer-by-layer fashion. Guo \emph{et al}.~\cite{guo2020model} further proposed to iteratively remove a group of channels from several selected layers instead of a single layer. Nevertheless, Guo \emph{et al}.~\cite{guo2020channel} observed the ``next-layer feature map removal'' problem that if a feature from the next layer will be removed at the next pruning stage, minimizing the reconstruction error of such features is unnecessary. To solve this, they considered both classification loss and feature importance as a pruning criterion to deal with the influence of removing next-layer feature maps.
There are also some approaches that judge the importance of channels using attention modules~\cite{yu2020antidote,yamamoto2018pcas,wang2019convolutional}. For example, Wang \emph{et al.}~\cite{wang2019convolutional} leveraged the attention module to obtain the scaling factors of each channels, which are then served as the importance criterion. Our approach differs from attention-based pruning methods in that it considers the class-wise contribution of a single channel on the basis of CNN interpretability, while attention based methods focus on the channel-wise attention scores.

The second group implements channel pruning in a training-adaptive manner by introducing extra sparsity regularization. For example, Huang \emph{et al}.~\cite{huang2018data} introduced a scaling factor to scale the outputs of specific structures and added sparsity on these factors. They then trained the sparsity-regularized mask for network pruning through data-driven selection. Luo \emph{et al}.~\cite{luo2020autopruner} employed an ``autopruner'' layer appended in the convolutional layer to prune filters automatically. By regularizing auxiliary parameters instead of original weights values, Xiao \emph{et al}.~\cite{xiao2019autoprune} pruned the CNN model via a gradient-based updating rule. Chen \emph{et al}.~\cite{chen2020dynamical} designed a channel-wise gate to dynamically estimate the conditional accuracy change and gradually prune channels during training process.
Tang \emph{et al}.~\cite{tang2021manifold} further explored the manifold information to dynamically excavate the channel redundancy of CNNs.

The last category of channel pruning approaches apply AutoML techniques to directly search the channel number of each layer under a given computation budget~\cite{he2018amc, liu2019metapruning}. For instance, He \emph{et al}.~\cite{he2018amc} proposed to automate the searching process by reinforcement learning. Liu \emph{et al}.~\cite{liu2019metapruning} trained a PruningNet in advance to predict the weights of candidate networks and leverage evolutionary algorithm to search for the bset candidate. Unfortunately, all of these methods conduct channel pruning with respect to the CNN output, failing to consider the internal mechanism of a CNN model. Though there are considerable improvements, interpretation for channel pruning remains an open problem.

\textbf{CNN Interpretation.}
Despite an impressive performance in various tasks, CNNs have long been known as ``Black-Box'' for its end-to-end learning strategy. As an Achilles' heel, CNN interpretability has attracted increasing attention in recent years~\cite{wu2017interpretable,yosinski2015understanding,zeiler2014visualizing,zhang2018interpretable,zhou2014object}. Most studies of understanding CNN representations fall into visualization. Zeiler \emph{et al}.~\cite{zeiler2014visualizing} proposed a visualizing technique by projecting the feature activations back to the input pixel space to observe the function of intermediate feature maps. Yosinkin \emph{et al}.~\cite{yosinski2015understanding} further developed a tool that visualizes the activations produced on each layer of a trained CNN model. They then gleaned several surprising intuitions using this tool, including that some channels always represent useful partial information for classification decisions like faces or text, although there are no explicit labels for these items. Such phenomenon is also found in~\cite{zhou2014object}, which motivates us to consider what kind of channels should be pruned from an interpretable perspective. Recently, CNN interpretation theory also demonstrates its effectiveness by achieving state-of-the-art results on various tasks such as image classification~\cite{zhang2018interpretable} and object detection~\cite{wu2017interpretable}. Hence we believe CNN interpretability could be a superior foreground for guiding channel pruning.

\section{Proposed Method}
\subsection{Background}\label{preliminary}
Considering an $L$-layer CNN model, its kernel weights can be represented as $\mathcal{W} = \{\mathcal{W}^1, \mathcal{W}^2, ..., \mathcal{W}^L\}$. The kernel in the $l$-th layer is denoted as $\mathcal{W}^l \in \mathbb{R}^{C^l_{out} \times C^l_{in} \times K^l \times K^l}$, where $C^l_{out} $, $C^l_{in}$, $K^l$ denote the numbers of output channels and input channels, and the kernel size, respectively. Let $ \mathcal{I}^{l} \in \mathbb{R}^{N \times C_{in}^l \times H^l \times W^l} $ be the input of the $l$-th layer where $N$ is the batch size of input images, and $H^l, W^l$ respectively stand for the height and width of the input. Given a batch of training image set $X$, associated with a set of class labels $Y^{N \times D}$ where $D$ represents the total number of categories in the training set, we denote $X = \mathcal{I}^1$. For the $i$-th input image $X_{i, :, :, :}$, we treat its label $Y_{i, :}$ as a one-hot vector. Sometimes we simply denote $Y_{i, :} = j$ to indicate that the $j$-th entry is set to one while the others are zero\footnote{It also indicates that image $X_{i, :, :, :}$ belongs to the $j$-th class.}. With a conventional CNN, the output of the $l$-th layer can be calculated as \footnote{The convolutional operation usually involves a bias term and is followed by a non-linear operation. For ease of representation, we omit them in this paper.}
\begin{equation}\label{conv}
\mathcal{O}^l = \mathcal{I}^l \circledast \mathcal{W}^l,
\end{equation}
where $\circledast$ denotes the convolutional operation. We have $\mathcal{O}^l = \mathcal{I}^{l+1} \in \mathbb{R}^{N \times C^{l+1}_{in} \times H^{l+1} \times W^{l+1}}$ and $C^{l}_{out} = C^{l + 1}_{in}$. Then, its training loss can be expressed as
\begin{equation}\label{original_loss}
\min_{\mathcal{W}} \; \mathcal{L}(X,\mathcal{W};Y).
\end{equation}

For channel pruning, a subgroup of output channels in $\mathcal{W}^l$ will be removed to obtain pruned kernel $\widehat{\mathcal{W}}^{l} \in \mathbb{R}^{\widehat{C}^l_{out} \times \widehat{C}^l_{in} \times K^l \times K^l}$ under the constraints of $\widehat{C}^l_{out} \le C^l_{out}$ and $\widehat{C}^l_{in} \le C^l_{in}$, thus it can reach a better trade-off between computation cost and accuracy performance. It is worth mentioning that the corresponding input channels of $\mathcal{W}^{l+1}$ are also removed. Accordingly, we can reformulate Eq.\,(\ref{conv}) and Eq.\,(\ref{original_loss}) in the pruned network as follows:
\begin{equation}\label{pruned_conv}
\widehat{\mathcal{O}}^l = \widehat{\mathcal{I}}^l \circledast \widehat{\mathcal{W}}^l,
\end{equation}
\begin{equation}\label{pruned_loss}
\min_{\widehat{{\mathcal{W}}}} \; \mathcal{L}(X,\widehat{\mathcal{W}};Y).
\end{equation}

As discussed in Sec.\,\ref{introduction}, most prior works perform channel pruning by directly judging channels' redundancy using an importance estimation or imposing sparsity penalty to dynamically conduct channel pruning. Essentially, these methods complete network pruning based on the final network outputs in an unexplainable way, which neglects the internal influence of channels, thus we term these methods ``Black-Box pruning.''

In contrast, for the first time, we conduct channel pruning by exploring the internal influence of CNNs. Our motivation is based on the observation in~\cite{yosinski2015understanding}, which reveals that the representations in the internal CNN layers are surprisingly local, implying that many channels are only responsible for extracting partial information. We argue that some of these partial information is redundant and may not be beneficial to classify all categories, as illustrated in Fig.\,\ref{visualization}. Hence, it is crucial to identify each channel's ability to derive recognizable local features that contribute to recognizing categories.

\subsection{Pipeline of White-Box}
In order to address the above issues, we propose a novel explainable channel pruning method, termed ``White-Box.''
The purpose of White-Box is to find channels that can generate feature maps containing discriminative category information as much as possible, so as to retain those that contribute to the recognition of most categories, and prune those channels that only benefit few categories.
Specifically, we firstly design a class-wise mask to multiply on each channel to guide a class-wise training.
When training a particular class of images, the corresponding mask will be activated for model inference and backpropagation, which will be described in detail in Sec.\,\ref{CMask}.
Fig.\,\ref{framework} shows the framework of White-Box for class-wise mask training. Subsequently,  as explained in Sec.\,\ref{IGV},  we propose a global voting mechanism to preserve those channels that make contributions to the recognition of most classes, as well as to automatically determine the layer-wise pruning rate without manual involvement.
Finally, a fine-tuning process is conducted to boost the performance of the pruned model.
Our White-Box is summarized in Alg.\,\ref{alg1}.
 %

\subsection{Class-wise Mask}\label{CMask}

\begin{algorithm}[t]
\caption{\label{alg1}Algorithm Description of White-Box}

\LinesNumbered
\KwIn{An L-layer CNN, global pruning rate $\alpha$, mask training epochs $\mathcal{T}$}
\KwOut{The pruned model and its parameters $\widehat{\mathcal{W}}$}
Initialize model parameters $\mathcal{W}$ , pruning rate $\widehat{\alpha} = 0$, class-wise mask $\mathcal{M} =  \{1\}$\;
\For{$h = 1 \rightarrow \mathcal{T}$}
{
	Train model with $\mathcal{W}$ and $\mathcal{M}$ via Eq.\,\ref{objectfunc}\;
}
Get criterion score $\mathcal{S}$ via Eq.\,\ref{score}\;
Sort S in ascending order\;
\While{$\widehat{\alpha} \leq \alpha$}
{
	Remove the first element in $\mathcal{S}$ and its corresponding channel in $ \mathcal{W}$ and mask in $\mathcal{M}$\;
	Update current FLOPs pruning rate $\widehat{\alpha}$\;
}
Integrate $\widehat{\mathcal{M}}$ into $\widehat{\mathcal{W}}$ via Eq.\,\ref{integrate} \;
Fine-tune pruned model to recover performance.
\end{algorithm}

The core of our White-Box is to assign per-layer kernel $\mathcal{W}^l$ a class-wise mask, which is formatted in the form of $\mathcal{M}^l \in \mathbb{R}^{D \times C_{out}}$. Specifically, the mask value $\mathcal{M}^l_{j, c}$ is built to measure the contributions of individual channels $\mathcal{W}^l_{c, :, :, :}$ to the network for recognizing the $j$-th category.

Then, for the $i$-th input image $X_{i, :, :, :}$ with label $Y_{i, :}$, the convolution using Eq.\,(\ref{conv}) in the forward propagation under our mask framework can be rewritten as
\begin{equation}\label{conv2}
\begin{split}
\mathcal{O}^l_{i,:,:,:} &=  \mathcal{I}^l_{i,:,:,:} \circledast (  \mathcal{M}^l_{Y_{i, :},:} \ast  \mathcal{W}^l), \;\; i = 1, 2,..., N,
\end{split}
\end{equation}
where $\ast$ denotes the channel-wise multiplication, that is, channel $\mathcal{W}^l_{c, :, :, :}$ is multiplied with the scalar mask $\mathcal{M}^l_{j, c}$. Subsequently, our training loss can be obtained as:
\begin{equation}\label{pruned_loss}
\min_{{\mathcal{W}}, \mathcal{M}} \; \mathcal{L}(X,\mathcal{W},
\mathcal{M};Y).
\end{equation}

The rationale of our mask design lies in that, during backpropagation, the mask $\mathcal{M}^l_{j, c}$ will receive the gradient signals regarding the input images of the $j$-th category. On the premise of this principle, if channel $\mathcal{W}^l_{c, :, :, :}$ benefits the network to recognize input images from the $j$-th category, $\mathcal{M}^l_{j, c}$ will be positively activated, and deactivated, otherwise. Therefore, our class-wise mask design can well reflect the internal logic in CNNs, which seamlessly follows our motivation behind the channel pruning in our white box.

In comparison with typical CNNs where the label information is utilized in the loss layer, our class-wise mask-based convolutional operations are more label-guided since it requires label information in every convolutional layer as shown in Eq.\,(\ref{conv2}). This poses a critical challenge of the over-fitting problem since the label information is in the format of one-hot vector, meaning that we need to provide ground-truth labels for each convolutional layer’s forward propagation. Such data flow during training varies largely from the real testing part, thus may cause the over-fitting problem.

Inspired by the label-smoothing regularization~\cite{szegedy2016rethinking},
we propose to solve this problem by softening the one-hot vector, denoted as $\widehat{Y} \in \mathbb{R}^{N \times D}$, element of which is defined as
\begin{equation}
    \widehat{Y}_{i, d} =
    \begin{cases}
    Y_{i, d}, & \mbox{if $Y_{i, d}$ = 1}, \\
    \mathcal{N}(0,1),   & \mbox{otherwise},
    \end{cases}
\end{equation}
where $\mathcal{N}(\cdot,\cdot)$ denotes the normal distribution.

Then, the convolution in Eq.\,(\ref{conv2}) is reformulated as
\begin{equation}\label{conv_soft}
\begin{split}
\mathcal{O}^l_{i,:,:,:} = \sum_{d=1}^D \mathcal{I}^l_{i,:,:,:} &\circledast \Big( \big( \widehat{Y}_{i,d} \cdot  \mathcal{M}^l_{d, :}\big) \ast  \mathcal{W}^l \Big), \\&i = 1, 2, ..., N.
\end{split}
\end{equation}

Thus, channel pruning can be realized by removing those channels with poor masks. It is natural to impose sparsity constraint on per-channel mask as
\begin{equation}\label{mask_sparsity}
    \min_{\mathcal{M}}\sum_{l=1}^L\sum_{c=1}^{C^l_{out}}\| \mathcal{M}^l_{:, c} \|_2.
\end{equation}

Note that we choose $\ell_2$-norm instead of $\ell_1$-norm as previous sparsity regularization works do~\cite{luo2020autopruner,xiao2019autoprune}. The rationale falls in that our object is not to regularize the masks to exactly 0s that $\ell_1$ norm leads to, but to measure the class-wise contribution of each channel. After training, we leverage global voting to directly obtain the pruned model, which will be introduced in the next section. Thus, we choose to leverage $\ell_2$-norm for its smoothness and rotation invariance.

Eq.\,(\ref{pruned_loss}) and Eq.\,(\ref{mask_sparsity}) lead to our final training loss:
\begin{equation}\label{objectfunc}
  \min_{\mathcal{W}, \mathcal{M}} \mathcal{L}(X, \mathcal{W}, \mathcal{M}; Y) + \lambda \sum_{l=1}^L\sum_{c=1}^{C^l_{out}}\| \mathcal{M}^l_{:, c} \|_2.
\end{equation}

Noticeably, the objective of Eq.\,(\ref{objectfunc}) targets at locating channels that contribute more to recognizing the input images, which then make up of the pruned kernel $\widehat{W}$ as described in Sec.\,\ref{preliminary}, followed by a series of fine-tuning procedures using loss objective of Eq.\,(\ref{pruned_loss}). Therefore, only a few epochs are needed to train our class-wise mask so as to derive $\widehat{\mathcal{W}}$ in our empirical observation\footnote{We consider 10\% of the total fine-tuning epochs for training the class-wise mask.}.

\subsection{Global Voting for Cross-layer Pruning}\label{IGV}

Given a global pruning rate $\alpha$, how to appropriately distribute it to each layer to preserve $C^l_{out}$ channels would significantly affect the performance of the pruned model~\cite{liu2019rethinking}. Prevalent methods resort to rule-of-thumb designs~\cite{li2017pruning, lin2020hrank} or complex structure search~\cite{liu2019metapruning, li2020eagleeye}.

Fortunately, our White-Box can tacitly obtain a global important criterion for all channels in the network and conduct layer-wise pruning rate decision in an iteratively-voting manner. Detailedly, considering a trained class-wise mask $ M^l_{:,c} \in \mathbb{R}^D$ of the $c$-th channel in the $l$-th layer, each item in this tensor represents this channel's ability for classifying one corresponding category of the dataset, thus we can measure this channel's contribution to overall classification performance by simply summing up these class-wise mask scores. We denote all scores of $\mathcal{M}^l$ as $\mathcal{S}^l \in  \mathbb{R}^{C^l_{out}}$:
\begin{equation}\label{score}
 \mathcal{S}_c^l = \sum_{d=1}^D M^l_{d,c},\; c = 1\;...\;C^l_{out},
\end{equation}
which then will serve as importance criterion for this channel.

Given a global pruning rate $\alpha$,  after obtaining all channels' importance scores $\mathcal{S}$ in the whole network, we iteratively remove the least-impact channels and calculate FLOPs pruning rate $\widehat{\alpha}$ of the current model until $\widehat{\alpha} \geq \alpha$. After voting, we integrate the left class-wise mask $\widehat{\mathcal{M}}$ into $\widehat{\mathcal{W}}$ to conduct fine-tuning for performance recovery. Particularly, as we soften the label obeying a standard normal distribution $\mathcal{N}(\mu = 0.5, \sigma = 1) $ during training except for the ground-truth related current input, the overall pruned $\widehat{\mathcal{M}}$ can be mixed into $\widehat{\mathcal{W}}$ by
\begin{equation}\label{integrate}
 \widehat{\mathcal{W}} =  \; \widehat{\mathcal{W}} \ast \sum_{d=1}^D \mu \;\widehat{M_{d,:}}.
\end{equation}

Lastly, more epochs are used to fine-tune the pruned model.

\section{Experiments}
\subsection{Implementation Details}
\textbf{Datasets and Backbones.} We conduct extensive experiments on two representative datasets including CIFAR-10~\cite{krizhevsky2009learning} and ILSVRC-2012~\cite{russakovsky2015imagenet} to demonstrate the efficacy of the proposed White-Box. We prune prevailing CNN models including VGG-16~\cite{simonyan2015very}, ResNet-56/110~\cite{he2016deep}, MobileNet-v2~\cite{sandler2018mobilenetv2} on CIFAR-10 and ResNet-50~\cite{he2016deep} on ILSVRC-2012.

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=0.9\linewidth,height=0.18\pdfpageheight]{different_rate.pdf}
\end{center}
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{0pt}
\vspace{-1.4em}
\caption{\label{vary_rate}Top-1 accuracy comparison between existing methods and the proposed White-Box under different pruning rates of FLOPs. The experiments are conducted using ResNet-56 and ResNet-110 on CIFAR-10.
\vspace{-1.8em}
}
\end{figure*}

\textbf{Configurations.}
We set the sparse parameter $\lambda$ as $10^{-2}$ for VGGNet-16 and MobileNet-v2, and $5 \times 10^{-4}$ for ResNets. Then, We train our class-wise masks using the original full network with a learning rate of 0.1 for 30 epochs on CIFAR-10 and 9 epochs on ILSVRC-2012. After the global voting, the pruned model is then fine-tuned via the SGD optimizer. The momentum, batch size are set to 0.9, 256, respectively, in all experiments.
On CIFAR-10, we iterate 300 epochs to fine-tune the pruned model \ with an initial learning rate of 0.1, which is divided by 10 at the 150-th and 225-th epochs.
On ILSVRC-2012, ResNet-50 is fine-tuned for 90 epochs with step scheduler learning rate, which begins at 0.1 and is divided by 10 every 30 epochs.
The weight decay rate is set to $5 \times 10^{-4}$ for all models except for MobileNet-v2: $4 \times 10^{-5}$  on CIFAR-10 and $10^{-4}$ on ILSVRC-2012.
All experiments are implemented with Pytorch~\cite{pytorch2015} and run on NVIDIA Tesla V100 GPUs.
The data argumentation includes crop and horizontal flip.

\begin{table}[!t]
\centering
\small
\caption{Results for pruning VGGNet-16 on CIFAR-10.}
\vspace{-0.5em}
\label{vgg_cifar10}
\setlength{\tabcolsep}{0.6em}
\begin{tabular}{cccc}
\toprule
 Model           &Top-1 Acc.& Acc.~$\downarrow$        & FLOPs $\downarrow$  \\

\midrule
$\ell_1$~\cite{li2017pruning}           &93.25\% $\rightarrow$ 93.40\%     & -0.25\%  &34.2\%   \\
GAL~\cite{lin2019towards}   &93.02\% $\rightarrow$ 92.03\%    &1.93\%       &39.6\%     \\
SSS~\cite{huang2018data}            &93.02\% $\rightarrow$ 93.02\%   &0.00\%         &41.6\%     \\
Slimming~\cite{liu2017learning} & 93.66\% $\rightarrow$ 93.80\% &-0.14\% & 51.0\% \\

HRank~\cite{lin2020hrank}     &93.02\% $\rightarrow$ 91.23\%  &1.79\%         &76.5\%  \\

\textbf{White-Box}           &~93.02\% $\rightarrow$ \textbf{93.47\%}& \textbf{-0.45\%}     &\textbf{76.4\%} \\

\bottomrule
\end{tabular}
\end{table}

\begin{table}[!t]
\setlength{\tabcolsep}{0.6em}
\centering
\small
\caption{Results for pruning ResNet-56 on CIFAR-10.}
\vspace{-0.5em}
\label{resnet56_cifar10}
\begin{tabular}{ccccc}
\toprule
Model           &Top-1 Acc.& Acc.~$\downarrow$       & FLOPs   $\downarrow$  \\
\midrule
HRank~\cite{lin2020hrank}             &93.26\% $\rightarrow$ 93.17\%     &0.09\%       &50.0\%     \\
AMC~\cite{he2018amc} & 92.80\% $\rightarrow$ 91.90\% &0.90\% &50.0\%  \\
SCP~\cite{kang2020operation} &93.69\% $\rightarrow$ 93.23\% &0.46\%  &51.5\%  \\
SFP~\cite{he2018soft} &93.59\% $\rightarrow$ 92.26\% &1.33\%  &52.6\%  \\
LFPC~\cite{he2020learning} &93.26\% $\rightarrow$ 93.24\% &0.02\%  &52.9\% \\
DSA~\cite{ning2020dsa} &93.12\% $\rightarrow$ 92.91\% & 0.21\% & 53.2\% \\
FPGM~\cite{he2019filter} & 93.59\% $\rightarrow$ 92.93\% &0.66\%&  53.6\% \\
\textbf{White-Box}                              &~93.26\% $\rightarrow$ \textbf{93.54\%}&\textbf{-0.28\%}& \textbf{55.6\%}     \\

\bottomrule
\end{tabular}

\end{table}

\subsection{Comparison on CIFAR-10}
We first demonstrate the superiority of White-Box on CIFAR-10. The FLOPs pruning rate of the compressed models and their top-1 accuracy performance are reported. The accuracy reported is in the format of ``pre-trained model $\rightarrow$ pruned model''. Several state-of-the-art (SOTA) channel pruning methods are compared, including $\ell_1$~\cite{li2017pruning}, SSS~\cite{huang2018data}, GAL~\cite{lin2019towards}, HRank~\cite{lin2020hrank}, SCP~\cite{kang2020operation}, DSA~\cite{ning2020dsa}, SFP~\cite{he2018soft}, FPGM~\cite{he2019filter}, LFPC~\cite{he2020learning}, Rethink~\cite{liu2019rethinking}, ABC~\cite{lin2020channel}, and WM~\cite{howard2017mobilenets}.

\textbf{VGGNet-16}.
Table\,\ref{vgg_cifar10} shows the results of pruning 16-layer VGGNet~\cite{simonyan2015very} model, which consists of 13 sequential convolutional layers and 3 fully-connected layers. As can be seen, White-Box yields significantly better top-1 accuracy of 93.47\% compared to the recent SOTA, HRank~\cite{lin2020hrank} of 91.23\%, under similar FLOPs reductions. Moreover, compared to GAL~\cite{lin2019towards} which simply imposes masks upon the outputs of convolutions, our White-Box that considers the internal influence of each channel to the categories, results in a significant reduction on the FLOPs, \emph{i.e.}, 76.64\% \emph{vs.} 39.6\%, while retaining a better top-1 accuracy of 93.47\% \emph{vs.} 92.03\%.

\textbf{ResNet}.
We also evaluate the network pruning performances of various methods on ResNet~\cite{he2016deep},a predominant deep CNN with residual modules, as shown in Table\,\ref{resnet56_cifar10} and Table\,\ref{resnet110_cifar10}. As can be observed, our White-Box increases the performance of original ResNet-56 by 0.28\% and removes around 55.60\% computation burden, while the other methods suffer the accuracy degradation more or less, even reducing less FLOPs. Besides, our White-Box also shows impressive superiority when pruning ResNet-110. With 66.0\% reductions on FLOPs, it still yields 0.55\% performance improvement, surpassing the other methods by a large margin.

\begin{table}[!t]
\setlength{\tabcolsep}{0.6em}
\centering
\small
\caption{Results for pruning ResNet-110 on CIFAR-10.}
\vspace{-0.5em}
\label{resnet110_cifar10}
\begin{tabular}{ccccc}
\toprule
Model           &Top-1 Acc.& Acc.~$\downarrow$       & FLOPs   $\downarrow$  \\
\midrule
$\ell_1$~\cite{li2017pruning}           &93.55\% $\rightarrow$ 93.30\%     & 0.25\%  &38.7\%   \\
Rethink~\cite{liu2019rethinking} &93.77\% $\rightarrow$ 93.70\%  &0.07\%           & 40.8\%  \\
SFP~\cite{he2018soft} &93.68\% $\rightarrow$ 93.38\% &0.30\%  &40.8\%  \\
GAL~\cite{lin2019towards}     &93.39\% $\rightarrow$ 92.74\%    &0.65\%     &48.5\%  \\
HRank~\cite{lin2020hrank}                       &93.50\% $\rightarrow$ 93.36\%  &0.14\%            & 58.2\%  \\
LFPC~\cite{he2020learning} &93.50\% $\rightarrow$ 93.07\% &0.43\%  &60.3\% \\
ABC~\cite{lin2020channel}      &93.57\% $\rightarrow$ 93.58\% &-0.01\%  &65.0\%  \\
\textbf{White-Box}    &~93.50\% $\rightarrow$ \textbf{94.12\%} &\textbf{-0.62\%} & \textbf{66.0\%} \\
\bottomrule
\end{tabular}

\end{table}

\begin{table}[!t]
\setlength{\tabcolsep}{0.6em}
\small
\centering
\caption{Results for pruning MobileNet-v2 on CIFAR-10.}
\vspace{-0.5em}
\label{mob_cifar10}
\begin{tabular}{cccc}
\toprule
 Model           &Top-1 Acc.& Acc.~$\downarrow$        & FLOPs $\downarrow$  \\

\midrule

WM~\cite{howard2017mobilenets}            &94.47\% $\rightarrow$ 94.02\%    &0.45\%           &27.0\%     \\

DCP~\cite{zhuang2018discrimination}            &94.47\% $\rightarrow$ 94.69\%    &-0.22\%          &27.0\%     \\

MDP~\cite{guo2020multi} 			&95.02\% $\rightarrow$ 95.14\%    &-0.12\%       &28.7\%     \\
\textbf{White-Box}                       &95.02\% $\rightarrow$ \textbf{95.28}\%& ~\textbf{-0.26\%}     &\textbf{29.2\%} \\

\bottomrule
\end{tabular}
\end{table}

\begin{table*}[!t]
\caption{\label{imagenet}\normalsize{ Results for pruning ResNet-50 on ILSVRC-2012.}}
\vspace{-0.5em}
\small
\centering
\setlength{\tabcolsep}{0.55em}
\begin{tabular}{ccccccc}
\toprule
Method   &Top-1 Acc. &Top-1 Acc. $\downarrow$ &Top-5 Acc. &Top-5 Acc. $\downarrow$ &FLOPs &FLOPs$\downarrow$ \\
\midrule
 SSS-26~\cite{huang2018data}&76.15\% $\rightarrow$ 74.18\%&1.97\% &92.96\% $\rightarrow$ 91.91\% &1.05\%  &2.82G &31.9\%    \\
 CP~\cite{he2017channel}&76.15\% $\rightarrow$ 72.30\%&3.85\% &92.96\% $\rightarrow$ 90.80\%  &2.16\%  &2.73G &34.1\%     \\
   SFP~\cite{he2018soft}  &76.15\% $\rightarrow$ 74.61\%&1.54\% &92.87\% $\rightarrow$ 92.06\% &0.81\%  &2.39G&41.8\%   \\
 GAL~\cite{lin2019towards}  &76.15\% $\rightarrow$ 71.95\%&4.20\% &92.96\% $\rightarrow$ 90.79\% &2.17\%  &2.33G&43.7\%   \\
 SSS-32~\cite{huang2018data}&76.12\% $\rightarrow$ 71.82\%&4.30\% &92.86\% $\rightarrow$ 90.79\% &2.07\%&2.33G &43.7\%  \\
HRank~\cite{lin2020hrank}&76.15\% $\rightarrow$ 75.01\% & 1.14\% &92.96\% $\rightarrow$  92.33\% & 0.63\% &2.30G & 43.9\%  \\
\textbf{White-Box} &76.15\% $\rightarrow$ \textbf{75.32\%}  &\textbf{ 0.83\%} &92.96\% $\rightarrow$ \textbf{92.43\%}  &~\textbf{0.53\%} &\textbf{2.22G}   &  ~\textbf{45.6\% } \\
MetaPruning~\cite{liu2019metapruning} & 76.60\% $\rightarrow$ 75.40\% &1.20\% & -& - & 2.00G & 48.7\% \\
FPGM~\cite{he2019filter} &76.15\% $\rightarrow$ 74.13\%  & 2.02\% &92.96\% $\rightarrow$ 92.87\% & 0.09\% &1.90G &  53.5\%   \\
 RRBP~\cite{zhou2019accelerate} &76.15\% $\rightarrow$ 73.00\%&3.15\% &92.96\% $\rightarrow$ 91.00\% &1.96\% &1.86G&54.5\% \\
 GAL~\cite{lin2019towards} &76.15\% $\rightarrow$ 71.80\%&4.35\% &92.96\% $\rightarrow$ 90.82\% &2.14\% &1.84G&55.6\% \\
 ThiNet~\cite{luo2017thinet} &72.88\% $\rightarrow$ 71.01\%&1.87\% &91.06\% $\rightarrow$ 90.02\%&1.12\%  & 1.71G&58.7\%   \\
 LFPC~\cite{he2020learning} & 76.15\% $\rightarrow$ 74.18\%   & 1.97\% &92.96\% $\rightarrow$ 91.92\%&  1.04\% &1.61G   & 60.8 \%  \\
 HRank~\cite{lin2020hrank} &76.15\% $\rightarrow$ 71.98\%&4.17\% &92.96\% $\rightarrow$ 91.01\% &1.95\% &1.55G &62.6\%   \\
 \textbf{White-Box}   &  76.15\% $\rightarrow$ \textbf{74.21}\%   & ~\textbf{1.94\%}  &~92.96\% $\rightarrow$ \textbf{92.01\% } & \textbf{ 0.95\%}& \textbf{1.50G}     & \textbf{  63.5\% } \\
\bottomrule
\end{tabular}
\end{table*}

\textbf{MobileNet-v2}. MobileNet-v2~\cite{sandler2018mobilenetv2} is a prevailing network with a compact design of depth-wise separable convolution. Due to its extremely small computation cost, pruning MobileNet-v2 becomes a particularly challenging task. Nevertheless, compared with the competitors in Tab.\,\ref{mob_cifar10}, White-Box still retains better top-1 accuracy of 95.23\%, while pruning more FLOPs of 29.2\%.

Further, we plot the performance comparison under different pruning rates of FLOPs in Fig.\,\ref{vary_rate}. To show our advantage, we compare the proposed White-Box with several SOTAs.
As can be observed, though the pruning rate changes, our White-Box consistently retains a higher top-1 accuracy, which well demonstrates the correctness of exploring the internal CNNs.

\begin{table}[!t]
\setlength{\tabcolsep}{0.8em}
\centering
\small
\caption{\label{speedup}FLOPs, latency and accuracy of White-Box for pruning the ResNet-50. Reported latency is the run-time of the corresponding network on one NVIDIA Tesla V100 GPU with a batch-size of 32.}
\vspace{-0.5em}
\begin{tabular}{ccccc}
\toprule
Method           &FLOPs & Latency&Speedup  &Top-1 Acc.   \\
\midrule
Baseline  & 4.11G  & 1.75ms  & 0.00$\times$  & 76.15\%   \\
White-Box &2.22G  & 1.29ms& 1.35$\times$  & 75.32\%   \\
White-Box  & 1.50G& 1.08ms& 1.62$\times$ & 74.21\%  \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Comparison on ILSVRC-2012}
We further show the results for pruning ResNet-50~\cite{he2016deep} on ILSVRC-2012. In Tab.\,\ref{imagenet},
Compared with the SOTAs, White-Box shows the best performance under different pruning rates. By setting $\alpha$ to 0.45, White-Box reduces the FLOPs to around 2.22B while obtaining the top-1 accuracy of 75.32\% and top-5 accuracy of 92.43\%. In contrast, the recent HRank~\cite{lin2020hrank} bears more computation of 2.30 FLOPs and poor top-1 accuracy of 75.01\%  and top-5 accuracy of  92.33\%. Further, we increase $\alpha$ to 0.63 and White-Box shows the least accuracy drops of 2.02\% in  top-1 accuracy  and 1.03\% in top-5 accuracy. With less FLOPs reductions, LFPC~\cite{he2020learning} shows poor top-1 accuracy of 74.18\% and top-5 accuracy of 91.92\%. Tab.\,\ref{speedup} reports the reduction of inference time. our White-Box achieves significant speedups while losing marginal performance. For instance, it obtains 1.62$\times$ GPU speedups with only 1.94\% top-1 accuracy drop, compared with the baseline.

\subsection{Ablation Study}\label{ablation}

\begin{table}[!t]
\setlength{\tabcolsep}{1em}
\arrayrulecolor{black}
\centering
\small
\caption{Top-1 accuracy comparison for pruning ResNet-56 with varyring configurations on CIFAR-10 under similar FLOPs pruning rate.}
\vspace{-0.5em}
\label{without_classwise}
\begin{tabular}{ccc}
\toprule
Setting           & Top-1 Acc.~$\downarrow$     &  FLOPs   $\downarrow$  \\
\midrule
\textbf{White-Box} & \textbf{-0.28\%} & \textbf{55.6}\% \\
w/o Class-wise mask& 1.43\% & 53.3\%\\
w/o Soft mask &  2.12\% & 54.7\%\\
w $\ell_1$-norm &0.89\% & 55.2\%\\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[!t]
\begin{center}
\includegraphics[width=1\linewidth,height=0.6\linewidth]{maskvalue.pdf}
\end{center}
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{0pt}
\vspace{-1.0em}
\caption{\label{mask_ablation}Mask values in varying layers after class-wise training for pruning ResNet-56 on CIFAR-10. The channel axis is sorted by the voting scores.
\vspace{-1.5em}
}
\end{figure}

\textbf{Class-wise Mask.}
In this section, we prune ResNet-56 and test its performance on CIFAR-10 as an example to investigate the influences of individual components in our class-wise mask.
We first train each channel with a single mask for all categories of images, denoted as w/o Class-wise mask in Table\,\ref{without_classwise}.
Such a mechanism suffers more accuracy drops as it fails to consider individual channel's discriminating power to recognize different categories as discussed in Sec.\ref{introduction}.
In addition, we conduct experiments without the smooth operation for mask activation, which is referred to as w/o Soft mask.
Table\,\ref{without_classwise} shows that such an implementation leads to the over-fitting problem that the network will converge in one epoch. Thus, the trained mask cannot well contribute to discriminating different categories, leading to an even worse top-1 accuracy than w/o Class-wise mask under a similar FLOPs reduction.

We also train the class-wise mask with $\ell_1$-norm regularization, denoted as ``w $\ell_1$-norm'' in Table\,\ref{without_classwise}. The $\ell_1$-norm brings worse performance that $\ell_2$-norm, thus demonstrating our motivation of choosing $\ell_2$-norm to measure the class-wise contribution of each channel. Lastly, we show the distribution of the mask value after training over different classes. Visualization in Fig.\ref{mask_ablation} shows that the class-wise mask value changes from channel to channel, which well confirms the motivation of White-Box. Channels with low mask values over all classes will be pruned using the global voting

\textbf{Sparsity factor.}
We further analyze the impact of the sparsity factor $\lambda$.
We choose to prune VGGNet-16 on the CIFAR-10 with different $\lambda$ under the similar FLOPs pruning rate.
In Fig.\,\ref{vgg_ablation}, with different $\lambda$, all of the pruned networks perform significantly better than the SOTA HRank~\cite{lin2020hrank}
To explain, our motivation for the class-wise mask training merely falls into observing each channel's contribution to classifying different categories of image, instead of recovering performance.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=1\linewidth,height=0.45\linewidth]{vgg_ablation.jpg}
\end{center}
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{0pt}
\vspace{-1.0em}
\caption{\label{vgg_ablation}Top-1 accuracy with different values of $\lambda$ for pruning VGGNet-16 on CIFAR-10 under similar FLOPs pruning rate. Best viewed in color.
\vspace{-1.5em}
}
\end{figure}

\section{Conclusion}
Based on visualization and analysis of the deep feature in CNNs, we proposed a new perspective of channel pruning for efficient image classification that one should preserve channels activating discriminative features for more categories in the dataset.
We further carry out channel pruning in a white box by devising a class-wise mask for each channel.
During training, different sub-masks are activated for model inference, \emph{w.r.t.} the current label of input images.
A global voting and a fine-tuning are then performed to obtain the compressed model.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}


\section{Introduction}\label{sec:introduction}

No one can say for sure when did music become a conscious part of human development, but undoubtedly, humans have been deliberately creating music for tens of thousands of years \cite{galpin2004howoldismusic}. This old art form has come a long a way to generate more than 50  billion U.S. dollars revenue in our modern age \cite{statista_music_revenue}.

Similar to many other domains, development of electronics and computers in the $20^{th}$ century has started a new era in the history of music. In this era, not only music-related tasks are being automated, but we are also witnessing new emerging tasks which were difficult to do, if not impossible, a few decades ago. For instance, in the past decade genre identification \cite{Doraisamy2008study}, music summarization \cite{mardirossian2006music}, music database querying \cite{eric2003name}, melodic segmentation \cite{pearce2008comparison}, and harmonic analysis \cite{chen2011music} are presented to the literature. Furthermore, algorithmic tools are used to model patterns of psycho-physical responses to music stimuli \cite{juslin2008emotional}, and the interaction between musical concepts and their interpretations in the brain \cite{krumhansl2001cognitive}. Moreover, using AI and Machine Learning, computers are now capable of generating new pieces of music which is never heard before \cite{dong2018musegan,Huang2018improved}.

Computational music algorithms need to accept music as an input, process it, and provide music as the output. The performance of these algorithms can be improved if we represent the music more effectively. An ineffective representation of music data can become the bottleneck of a system, for instance, the existing generative models struggle to generate music for a large orchestra \cite{guan2019gan}. As it will be discussed in Section~\ref{sec:representation}, the existing representations are developed with pure computational concepts and ignoring high-level music theory. Therefore, the need for a representation which supports the application of high-level music theory concepts is significant. In this paper, we address the disjoint of music theory and computational music by developing an open-source representation tool based on music theory. We then use our tool for studying and analyzing the trends in different composers’ works.

The rest of the paper is organized as follows: Common approaches in representing music and their pros and cons are described in Section~\ref{sec:representation}. Section~\ref{sec:interval} explains intervals in music theory, as a baseline for the solution provided in Section~\ref{sec:music_embedding}. Finally, we demonstrate an example use case of the developed solution in Section~\ref{sec:usecase}, and we conclude the paper in Section~\ref{sec:conclusion}.

\section{Music representation}\label{sec:representation}
In this section we discuss the common practices in representing and storing music in digital media. There are two main categories for data representation in music: audio and symbolic.

\subsection{Audio representation}
Music is an audio which satisfies some specific characteristics and criteria. Therefore, it is reasonable to represent and store music using general audio formats such as WAV and MP3. An analogy to this approach is to store text data in a raster image file, e.g., JPEG; the result can be interpreted and read by human, but the text data is lost. While this approach works great for recording, it does not provide any information about the structure of the music. Therefore, it is not suitable for applications which require data regarding the structure of the music.

\subsection{Symbolic representation}\label{subsec:symbolic}

In contrast to audio representation, symbolic representation stores the structure of the music itself rather than the audio signal which is generated by performance. There exist different approaches for representing music symbolically, each of which providing different level of abstraction. The common theme in these approaches is that music can be seen as a sequence of events. For instance, events in a piece for piano describe when and how hard each piano key should be pressed and how long it should be held. The most common approaches in symbolic music representation are described in the following.

\subsubsection{Music notation}
Music notation is the most human-readable and the oldest representation system. In a standard western notation, music is written using five parallel horizontal lines which are called staff (stave). These five lines represent changes in the pitches, where moving to higher lines increases the pitch. A symbol, called clef, is required to define origin of the pitch and assign notes to lines. The symbols are written and read from left to right and from top to bottom. The shape of each note determines the duration of that note. Usually, vertical lines, called bar lines, intercept staff lines at regular intervals to form bars. A symbol, called time signature, consisting of two numbers describes how many beats each bar represents.

Music notation was developed and improved with the music itself. Although some scorewriter programs, such as MuseScore and Finale, are developed to create and edit files using music notation, this writing system is designed for human; computers cannot run algorithms efficiently using music notation.

\subsubsection{**kern}
**kern is a representation scheme employed in Humdrum toolkit which is  a set of resources for computational music analysis \cite{Huron2002music}. **kern offers a one-to-one mapping to the basic music notation symbols. **kern can be used to encode pitch and duration as well as common score-related information. In other words, **kern is a text version of music notation, with some limitations.

\subsubsection{ABC notation}
Similar to **kern, ABC notation is a text-based encoding scheme for music notation. ABC notation has less vocabulary than **kern and supports fewer music symbols, but it is easier to read and understand by human.

\subsubsection{MusicXML}
As its name suggests, MusicXML is a music interchange language based on Extensible Markup Language (XML) \cite{muiscXML}. MusicXML inherits pros and cons of XML, namely, MusicXML offers openness, extensibility, self-description, content and presentation separation, while it is verbose and redundant, and more difficult for human to read and understand compared to **kern and ABC notation.

\subsubsection{MIDI standard}
Musical Instrument Digital Interface (MIDI) is the most popular standard for storing and transferring music  \cite{midi}. MIDI standard serves as a storage and retrieval protocol as well as a communication protocol that connects electronic musical instruments, computers, and related audio devices for playing, editing, and recording music. MIDI protocol works based on MIDI messages. A MIDI message describes an event, for instance, a \textit{note on} message signals that a particular note on a particular channel has to sound at the given time with the specified velocity. Due to its binary nature, MIDI is not human-readable, but it is effective in storing and communicating symbolic music between digital interfaces.

\subsubsection{Pianoroll}\label{subsec:pianoroll}
Pianoroll was invented in late $19^{th}$ century as a mechanical mean to store music \cite{welte1883Mechanical}. The general theme for mechanical pianorolls was similar to punched cards: the storage medium is moving under a series of pistons, the holes in the medium make the pistons move which in turn makes the music instrument (generally a piano or a church organ) sound.

With the emergence of Digital Audio Workstations (DAW), pianoroll became digitized. A mechanical pianoroll can be described as a 2D array, in which one dimension represents time while the other one represents notes. The element in the $i^{th}$ row and $j^{th}$ column describes the behavior of the $i^{th}$ note at the $j^{th}$ timestep, where zero means silence and any positive number indicates the velocity of the note.

Pianoroll is the most popular representation scheme in digital music. Popular DAWs such as FL Studio and Cubase only provide pianoroll representation. However, computationally speaking, pianorolls are heavily sparse. This makes it difficult and inefficient to run algorithms using pianorolls.

\section{Intervals in music theory}\label{sec:interval}

As discussed earlier, there is a disjoint between music theory and representation approaches. We aim to close this gap by developing a toolkit based on music theory which can be used to run algorithms efficiently. Since intervals are the building blocks of music theory \cite{piston1948harmony}, it is reasonable to build this toolkit based on intervals. In this section we discuss the intervals while the next section is devoted to the technical details of the tookit.

Music quantizes the sound in both time and frequency domains. In the frequency (pitch) domain, the quantum value is called a semitone. Regardless of different systems to assign a frequency to each semitone (i.e., tuning), everything is measured in terms of semitones. Interval is an inclusive distance measure based on semitones; the relation between two notes is analyzed based on their interval. Interval is the building block of music theory. Almost all high-level musical concepts are built and defined based on intervals. An interval is a multi-dimensional measure, consisting of the following information:
\begin{enumerate}
    \item An ordinal number (usually less than $8^{th}$) which depends on the name of the notes, e.g., A to A is first, A to B is second, B to G is sixth.
    \item A mode/type usually from the set, \{diminished, minor, perfect, major, augmented\}, which depends on the note modifiers, e.g., C to Db is min $2^{nd}$, C to D is maj $2^{nd}$, C to D\# is aug $2^{nd}$.
    \item A direction: ascending or descending (if the second note has a higher pitch it is ascending and vice versa), e.g., C to E is maj $3^{rd}$ asc while E to C is maj $3^{rd}$ des.
    \item Compound versus simple interval. Notes go from A to G then cycle back to A. Two different notes with the same name are called octaves (because they form the $8^{th}$ interval). If an interval is larger than an octave, it is compound and has the same functionality/properties of its corresponding simple interval. For instance, intervals $3^{rd}$ and $10^{th}$ are almost equal.
\end{enumerate}

\begin{table}[tpb]
  \caption{Q-Table for converting semitones to intervals and vice versa}
\centering
  \begin{tabular}{ccccc}
  \toprule
  &\multicolumn{2}{c}{\bfseries Music Theory}& \multicolumn{2}{c}{\bfseries Music Embedding}\\
  \cmidrule(r){2-3} \cmidrule(r){4-5}
  \bfseries Semitones & \bfseries Order & \bfseries  Type  & \bfseries Order & \bfseries Type\\
  \midrule
    0&First&Perfect&1&0\\
     1&Second&minor&2&-1\\
     2&Second&Major&2&1\\
     3&Third&minor&3&-1\\
     4&Third&Major&3&1\\
     5&Fourth&Perfect&4&0\\
     6&Fifth&Diminished&5&-2\\
     7&Fifth&Perfect&5&0\\
     8&Sixth&minor&6&-1\\
     9&Sixth&Major&6&1\\
     10&Seventh&minor&7&-1\\
     11&Seventh&Major&7&1\\
  \bottomrule
  \end{tabular}
\label{tab_q_table}
\end{table}

Different intervals might have the same number of semitones, for instance, aug $4^{th}$ and dim $5^{th}$ both have 6 semitones. The decision between alternative naming is made by taking the scale and alternation into account, which needs expert opinion. The most reasonable alternatives for translating semitones to and from intervals are provided in Table~\ref{tab_q_table}. As mentioned before, compound intervals can be broken into a simple interval plus some octave offset. For this reason, Table~\ref{tab_q_table} does not have a row for 12 semitones because it equals octave which can be encoded as perfect $1^{st}$ plus one octave offset.

\section{Music embedding toolkit}\label{sec:music_embedding}

The proposed Music embedding toolkit is developed for Python. Its source code \footnote{Link to music embedding toolkit source code: \url{https://github.com/PooyaHekmati/music_embedding}} and documentation\footnote{Link to music embedding toolkit documentation: \url{https://pooyahekmati.github.io/music_embedding/}} are publicly available. Additionally, this toolkit is published through Python Package Index and can be accessed with:

\begin{Verbatim}[commandchars=\\\{\}]
\PYG{n}{pip} \PYG{n}{install} \PYG{n}{music\char`\_{}embedding}
\end{Verbatim}

\begin{comment}

\begin{minted}{python}
pip install music_embedding
\end{minted}

\end{comment}
This Python package is developed based on intervals in the music theory. Hence, it has a class called \textit{interval} which can be used to have interval objects. Another class, called \textit{embedder}, is developed based on the \textit{interval} class. The \textit{embedder} class in the \textit{Music Embedding} package provides functionality to convert pianorolls to and from sequences of intervals with the different configurations. It also provides useful utilities such as extracting melody from a pianoroll, applying Run-Length Encoding (RLE) to a sequence of intervals, and bulk processing.

As explained in Section~\ref{sec:interval}, an interval is the relation between two notes with respect to their pitches; yet, this does not explain how those two notes are related to each other in the time domain. While the notes in an interval can be in any point of time, two special cases are famously known as \textit{melodic interval} and \textit{harmonic interval}. A \textit{melodic interval} is when the second note immediately follows the first note. A \textit{harmonic interval} is when the two notes sound together at the same time. A sequence of \textit{melodic intervals} creates \textit{melody} while a sequence of \textit{harmonic intervals} creates \textit{harmony}. In addition to \textit{melodic} and \textit{harmonic intervals}, the relation between each note and the first note of the same bar is also important. This is because the most strong beat in a bar is usually the first beat and the note at the first beat is of significance. The \textit{embedder} class supports all of these three configurations.

Figure~\ref{fig_orchnet_example_encoded} illustrates the melodic embedding of the piece in Figure~\ref{fig_orchnet_example}. The resolution per beat for this example is 24; therefore, each bar has 96 time steps and the entire piece has 384 time steps. In Figure~\ref{fig_orchnet_example_encoded}, consecutive columns which have the same value are summarized into a single column using RLE. This can be interpreted as note duration. The following code snippet is used to generate Figure~\ref{fig_orchnet_example_encoded}.

\begin{Verbatim}[commandchars=\\\{\}]
\PYG{k+kn}{import} \PYG{n+nn}{music\char`\_{}embedding}
\PYG{k+kn}{import} \PYG{n+nn}{pypianoroll}

\PYG{k}{if} \PYG{n+nv+vm}{\char`\_{}\char`\_{}name\char`\_{}\char`\_{}} \PYG{o}{==} \PYG{l+s+s1}{\char`\'{}\char`\_{}\char`\_{}main\char`\_{}\char`\_{}\char`\'{}}\PYG{p}{:}
    \PYG{n}{midi\char`\_{}path} \PYG{o}{=} \PYG{l+s+sa}{r}\PYG{l+s+s1}{\char`\'{}c:\char`\\{}music embedding example.mid\char`\'{}}
    \PYG{n}{multi\char`\_{}track} \PYG{o}{=} \PYG{n}{pypianoroll}\PYG{o}{.}\PYG{n}{read}\PYG{p}{(}\PYG{n}{midi\char`\_{}path}\PYG{p}{)} \PYG{c+c1}{\char`\#{}Open the MIDI file using pypianoroll}
    \PYG{n}{embedder} \PYG{o}{=} \PYG{n}{music\char`\_{}embedding}\PYG{o}{.}\PYG{n}{embedder}\PYG{o}{.}\PYG{n}{embedder}\PYG{p}{()} \PYG{c+c1}{\char`\#{}Create embedder object}
    \PYG{n}{melodic\char`\_{}intervals} \PYG{o}{=} \PYG{p}{[]}
    \PYG{k}{for} \PYG{n}{i} \PYG{o+ow}{in} \PYG{n+nb}{range}\PYG{p}{(}\PYG{n+nb}{len}\PYG{p}{(}\PYG{n}{multi\char`\_{}track}\PYG{o}{.}\PYG{n}{tracks}\PYG{p}{)):} \PYG{c+c1}{\char`\#{}Get melodic intervals for each track}
        \PYG{n}{melodic\char`\_{}intervals}\PYG{o}{.}\PYG{n}{append} \PYG{p}{(}\PYG{n}{embedder}\PYG{o}{.}\PYG{n}{get\char`\_{}melodic\char`\_{}intervals\char`\_{}from\char`\_{}pianoroll} \PYG{p}{(}
            \PYG{n}{pianoroll} \PYG{o}{=} \PYG{n}{multi\char`\_{}track}\PYG{o}{.}\PYG{n}{tracks}\PYG{p}{[}\PYG{n}{i}\PYG{p}{]}\PYG{o}{.}\PYG{n}{pianoroll}\PYG{p}{))}
    \PYG{n}{melodic\char`\_{}rle} \PYG{o}{=} \PYG{n}{embedder}\PYG{o}{.}\PYG{n}{get\char`\_{}RLE\char`\_{}from\char`\_{}intervals\char`\_{}bulk}\PYG{p}{(}\PYG{n}{melodic\char`\_{}intervals}\PYG{p}{)} \PYG{c+c1}{\char`\#{}Get RLE}
\end{Verbatim}

\begin{comment}

\begin{minted}{python}
import music_embedding
import pypianoroll

if __name__ == '__main__':
    midi_path = r'c:\music embedding example.mid'
    multi_track = pypianoroll.read(midi_path) #Open the MIDI file using pypianoroll
    embedder = music_embedding.embedder.embedder() #Create embedder object
    melodic_intervals = []
    for i in range(len(multi_track.tracks)): #Get melodic intervals for each track
        melodic_intervals.append (embedder.get_melodic_intervals_from_pianoroll (
            pianoroll = multi_track.tracks[i].pianoroll))
    melodic_rle = embedder.get_RLE_from_intervals_bulk(melodic_intervals) #Get RLE
\end{minted}

\end{comment}

 \begin{figure}
 \centering
 \includegraphics[width=\textwidth]{Pictures/Orchnetexample.png}
  \caption{A sample piece for string quartet}
  \label{fig_orchnet_example}
 \end{figure}

\begin{figure}

\centering
\includegraphics[width=\textwidth]{Pictures/Orchnetexampleencodedmelody.png}
\caption{The encoded representation of Figure~\ref{fig_orchnet_example} based on intervals}
\label{fig_orchnet_example_encoded}
\end{figure}

\section{Use cases}\label{sec:usecase}

The music embedding toolkit can be used in numerous applications, including but not limited to statistical and probabilistic analysis of music pieces, developing generative models to have AI-synthesized music, genre classification,
mood recognition, melody extraction, audio-to-score alignment, and score structure analysis. In this section, we explore one simple use case of the music embedding toolkit by analyzing the trend in different composers' works.

We perform the analysis based on a dataset on Kaggle\footnote{Link to the classical dataset on Kaggle: \url{https://www.kaggle.com/blanderbuss/midi-classic-music/}}. This MIDI dataset contains 3,920 pieces from 144 classical composers. After extracting all MIDI files from the zip files in the dataset, we checked the authenticity of the files by randomly selecting files ($n=100$) and comparing them to verified sources. We then performed data cleaning using Pypianoroll\cite{pypianoroll} and pretty\_midi \cite{pretty_midi} packages. The following criteria were considered for excluding a file:
\begin{enumerate}
  \item if either Pypianoroll or pretty\_midi cannot handle the file and throw an error,
  \item if the file has more than one time signature change event,
  \item if the file contains less than 10 bars (the number of bars was calculated as: \\$active\ length/beat\ resolution/time\ signature\\ numerator$),
  \item if the numerator of the first time signature event is 1.
\end{enumerate}

\begin{figure}
  \centering
  \includegraphics[width=0.75\columnwidth]{Pictures/RadarChart.png}
  \caption{Radar chart for the top 15 composers, minor-to-Major and Descending-to-Ascending ratios}
\label{fig_radar}
\end{figure}

Figure~\ref{fig_radar} shows a radar chart in which 15 composers with the most data in the dataset are compared with each other and the entire dataset. In this chart, the ratio of occurrences of minor and Major intervals are compared both harmonically and melodically. Also, the ratio of descending and ascending melodic intervals is shown. The minor-major ratios are calculated by:
\begin{equation}
r=\frac{f_{min}}{f_{min}+f_{Maj}}
\end{equation}
where $r$ is the ratio and $f_{min}$ and $f_{Maj}$ are the occurrences of minor and Major intervals respectively. Similarly, the ratio of descending and ascending melodic intervals is calculated by:
\begin{equation}
r=\frac{f_{dsc}}{f_{dsc}+f_{asc}}
\end{equation}
where $r$ is the ratio and $f_{dsc}$ and $f_{asc}$ are the occurrences of descending and ascending intervals respectively.

The minor-Major ratio for harmonic intervals in the entire dataset is 0.5011, which indicates both minor and Major harmonic are equally used in general. Busoni and Tchaikovsky have the highest ratios with 0.5265 and 0.5164 respectively. The lowest ratios belong to Chopin and Bizet with 0.4753 and 0.4807 respectively. This means that the popularity of minor and Major harmonic intervals is almost equal and does not deviate from the general trend from composer to composer.

The minor-Major ratio for melodic intervals in the entire dataset is 0.4773, which is lower than the same value for harmonic intervals. Brahms and Dvorak have the highest melodic ratios with 0.5343 and 0.5273 respectively. The lowest ratios belong to Haendel and Tchaikovsky with 0.4428 and 0.4431 respectively. Although the deviations in the melodic ratios are more than the deviations in the harmonic ratios, still they are within the same range. However, the melodic ratios of only Schubert, Brahms, Chopin, and Dvorak are greater than their harmonic ratios while for the rest of composers and the entire dataset the opposite is true.

The descending to ascending ratio for the entire dataset is 0.4030 which shows that the general trend is to use ascending intervals more often. Since the range of notes is limited, upward and downward movements need to cancel each other out to ensure the notes are in the feasible range. Having a descending to ascending ratio which is far from 0.5 concludes that ascending intervals are generally smaller than descending intervals; therefore, more ascending intervals are needed to compensate. In contrast to Bach who has the highest ratio (0.4611), Tchaikovsky has the lowest ratio (0.2351). This means that on average, out of every four intervals, Tchaikovsky has used only one descending interval.

As explained before, the sequence of intervals makes a piece sound how it sounds. Therefore, analyzing how intervals follow each other is of interest. Figure~\ref{fig_melodic_chord} shows the trends in each pair of melodic intervals happening next to each other. This figure is generated for the entire dataset and Mozart, Bach, and Beethoven as they have the highest number data points in the dataset. Figure~\ref{fig_melodic_chord} shows that unlike Mozart and Beethoven, Bach has used unison and octave intervals less frequently. Also, in all four diagrams, dim $5^{th}$ has the lowest value. Furthermore, the strongest connection is consistently between Maj $2^{nd}$ and min $7^{th}$. A similar figure, but for harmonic intervals is shown in Figure~\ref{fig_harmonic_chord}. Consonant and dissonant intervals can clearly be seen in this figure. Maj $7^{th}$ and min $2^{nd}$ are strongly dissonant in music theory and in Figure~\ref{fig_harmonic_chord} we see how small their proportions are. In contrast, perfect intervals, which are considered to be consonant, have a dominant presence.

\begin{figure}[b]
\centering
\includegraphics[width=\textwidth]{Pictures/MelodicChordDiagram.png}
\caption{Occurrence of pairwise melodic intervals}
\label{fig_melodic_chord}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{Pictures/HarmonicChordDiagram.png}
\caption{Occurrence of pairwise harmonic intervals}
\label{fig_harmonic_chord}
\end{figure}

\section{Conclusion}\label{sec:conclusion}

There is no doubt that representing data differently can affect efficiency and performance of algorithms. There exist different representation schemes and systems for symbolic music; however, they are developed for storage rather than a communication medium which can be used to let the computers know how humans think when composing or analyzing a musical piece. This shortcoming has slowed down the research and development of computational music domain.

In this paper, we tried to address this gap by presenting the music embedding toolkit which is a Python package based on music theory. This toolkit is developed based on intervals in music theory and offers multiple tools to encode and decode pianorolls to and from sequences of intervals. We showed one potential application of this toolkit by running a numerical analysis on classical pieces of music.

\IfFileExists{\jobname.ent}{
   \theendnotes
}{

}

\bibliographystyle{unsrt}