Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN’s size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels. We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1–7.3× convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers.
1 INTRODUCTION
Due to the success of deep neural networks in a broad set of practical and even critical artificial intelligence tasks, they are now widely deployed in a spectrum of platforms: smart phones, autonomous cars, data center servers, and even supercomputers. While suitably designed and trained CNNs can be powerful, they are often large – requiring many parameters (e.g., the celebrated AlexNet (Krizhevsky et al., 2012) has 60 millions). That large neural network models incur cost in terms of memory, energy, and inference speed is easy to see.
This motivated a line of research (Han et al. (2015; 2016b); Guo et al. (2016); Denton et al. (2014), to name a few) that tries to prune the parameters after a CNN design is trained and proved useful. A common thread is to post-process a trained CNN. Post-processing may consist of retraining with sparsity inducing regularization or of approximating tensors of parameters via tensor factorization. These methods reduce the size of CNNs significantly while preserving inference accuracy. Nevertheless, the inference speed gains in pruned networks is not nearly as impressive as the size reduction. In this sense, the benefits of CNN pruning seem not fully realized.
While seemingly unintuitive, that the significantly pruned CNNs run not nearly as significantly faster can be easily explained. First, fully connected (fc) layers usually contain the bulk of the parameters
while convolutional (conv) layers consume the bulk of computation time. This property shows that reducing the size of just the fc layers will readily lead to meaningful reduction in size as in Han et al. (2016b); Guo et al. (2016); but little speed improvement.
The crux of speed improvement thus lie in actual fast convolution of sparse kernels with feature maps (not just floating-point operations reduction), which is a challenging problem. It is well known in the field of numerical linear algebra that the performance of sparse matrix operations is typically memory bandwidth bound. Direct application of the sparse matrix operations to compute the conv layers when the kernels are sparse will likely result in sub-optimal speed gains. This concern on low efficiency of sparse operations is also discussed in the design of GoogLeNet (Szegedy et al., 2015). We will term methods that work directly with sparse data structures “sparse methods.” Alternative to sparse methods, “dense methods” gather data in a way that allow the actual convolution be performed by dense linear algebra functions such as GEMM. An example is found in (Lebedev & Lempitsky, 2015; Wen et al., 2016) which produces some group-wise sparsity patterns that facilitate the use of existing and highly tuned dense matrix computation library functions to perform the convolutions. However, imposing sparsity patterns limits the sparsity level that would otherwise be achievable had arbitrary patterns been allowed. We note that high compression in the conv layers are gaining importance as these layers consist a much larger percentage of parameters in recent networks such as GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2015).
We view sparse methods differently. Convolutions in CNNs involve multiple channels and thus offer much higher data reuse than typical sparse matrix operations in scientific computing. Specifically, we present a highly efficient direct sparse convolution design formulated as sparse-matrix-dense-matrix multiplication with the dense matrix columns generated on-the-fly from a single column vector. In addition to being highly efficient, this sparse convolution design is friendly with convolution kernels with arbitrary sparsity patterns. We call this element-wise sparsity to distinguish it from group-wise sparsity mentioned previously. As shown later on, accepting element-wise sparsity significantly increases the achievable sparsity level.
Complementing our sparse convolution design, we formulate a performance model to elucidate when and how best to use sparse convolutions on different computer architectures and at different CNN layers. Our formulation follows the roofline model (Williams et al., 2009). In particular, our model suggests (correctly) that sparse convolution can improve inference speed even with a moderate sparsity level of around 70%. In addition, the model provides upper and lower bounds of sparsity levels that can contribute to speed improvements. Sparsity higher than the upper bound offer no further speed improvement; and sparsity lower than the lower bound can in fact slow down inference rather than accelerating it.
Combining the sparse convolution design with the performance model allows us to prune a CNN in a co-design manner, with our proposed new pruning algorithm—Guided Sparsity Learning (GSL). As illustrated later, we can adjust sparsity targets precisely at different layers so as to maximize inference speed, best preserve accuracy, and minimize a network’s size. In some cases, particular layers are identified as best not pruned at all due to no potential speedups, leaving them unchanged gives other layers more room for gainful pruning in terms of size and speed.
Our paper makes the following contributions:
• A high performance sparse convolution design that takes advantage of arbitrary sparsity patterns and outperforms dense convolution even with a moderate sparsity.
• A general performance model that (1) projects speedups over dense convolutions on varying level/types of sparsity and different computing platforms and (2) provides training guidelines for precisely targeting layers and sparsity ranges with potential to accelerate inference.
• Guided Sparsity Learning (GSL), the first pruning algorithm fusing the awareness of speedup potential into sparsity learning; and its application to AlexNet and GoogLeNet. In particular, in GoogLeNet, we prune out more than 80% of parameters of all 5×5/3×3 conv layers and fc layers with no accuracy drop.
• An optimized sparse convolution implementation (http://github.com/IntelLabs/SkimCaffe) that provides 7.3×, 3.4×, 3.1× speedups of convolution layers in AlexNet over dense methods on Intel Atom, Xeon, and Knights Landing processors, respectively, with no accuracy drop. In particular, this paper is one of the first evaluations of Xeon Phi processors on deep learning algorithms.
for each output channel n { for j in [W.rowptr[n], W.rowptr[n+1]) { off = W.colidx[j]; coeff = W.value[j] for (int y = 0; y < H_OUT; ++y) { for (int x = 0; x < W_OUT; ++x) { out[n][y][x] += coeff*in[off+f(0,y,x)]
} }
} }
Figure 2: Sparse convolution pseudo code. Matrix W has compressed sparse row (CSR) format, where rowptr[n] points to the first non-zero weight of nth output channel. For the jth nonzero weight at (n,c,r,s), W.colidx[j] contains the offset to (c,r,s)th element of tensor in, which is pre-computed by layout function as f (c,r,s). If in has CHW format, f (c,r,s) = (cHin + r)Win + s. The “virtual” dense matrix is formed on-the-fly by shifting in by (0,y,x).
The rest of the paper is organized as follows. Section 2 presents the details of our sparse convolution design, formulation of the performance model, the Guided Sparsity Learning (GSL) pruning algorithm, and how they are combined to prune and accelerate CNNs. Section 3 demonstrates the effectiveness of these developments on AlexNet and GoogLeNet on a variety of platforms. Section 4 discusses related works and review the state of the art. Section 5 concludes and outlines a few next-steps.
2 GOING FASTER WITH DIRECT SPARSE CONVOLUTION, PERFORMANCE MODEL, AND GUIDED PRUNING
As explained previously, prunning CNN models does not yet benefit inference speed as much as model size reduction. This section first presents our efficient direct sparse convolution design that remedies this situation significantly. We then develop a performance model that projects speedup over different sparsity levels and on different processor architectures. The model guides our speedup-aware training method, Guided Sparstiy Learning (GSL).
2.1 DIRECT SPARSE CONVOLUTION
A sparse convolution for the all output positions across all output channels can be eventually considered as a virtual sparse-matrix-dense-matrix multiplication (SpMDM), as described in the following. Consider a bank of N filters each with size R×S against an Hin×Win feature with C input channels. We denote the filter bank as a 4-mode tensor W with size N×C×R× S, the input feature as a 3-mode tensor I with size C×Hin×Win, and the output feature as a 3-mode tensor O with size N×Hout ×Wout . The output value at (y,x)th position of nth output channel is computed by
O(n,y,x) = C−1
∑ c=0
R−1 ∑ r=0 S−1 ∑ s=0 W (n,c,r,s)I (c,y+ r,x+ s), (1)
which is a dot-product of two 3D tensors as shown in Figure 1. This can be treated as a vector dot-product by: first vectorizing the 3D subtensor of W corresponding to the nth output channel, then vectorizing I (denoted as vec(I )), and finally stretching the first vector to match the dimension of two vectors. When W is sparse, then this vector dot-product becomes a sparse-vector-dense-vector dot-product. Consider flattening dimensions except the first one of W into a sparse matrix W(1) (i.e. mode-1 matricization of W as in Kolda & Bader (2009)), with its row vectors stretched to match the dimension of vec(I ). O(n,y,x) is then the dot-product between the nth row of W(1) and vec(I ). Subsequently, the values at the same given (y,x)th position of all N output channels can be computed
collectively as a sparse-matrix-dense-vector multiplication (SpMV):
O(1) (:,yWout + x) = W(1) · vec(I y,x) , (2)
where I y,x denotes the tensor I with its last two dimensions shifted by (y,x). The values at different output positions can be computed as a sparse-matrix-dense-matrix multiplication (SpMDM), where the columns of the dense matrix are actually the same vector vec(I ) but with different offsets. In order to save bandwidth usage, we operate with a virtual dense matrix, Ivirtual, where its columns are generated on the fly by adjusting indices through which we access vec(I ).
Using the virtual dense matrix essentially skips the lowering step used in standard frameworks such Caffe, and, therefore, we call our method direct sparse convolution, to distinguish it from sparse convolution with lowering such as the method used in Liu et al. (2015). The lowering approach replicates the input feature multiple times, significantly reducing arithmetic intensity. The lowering process has demonstrated overhead for dense convolution as in Hadjis et al. (2015); Chintala (2015), and is particularly problematic for sparse convolution with intensity already lower than its dense counter part. Figure 3 demonstrates the advantage of our direct sparse convolution, using the performance model that will be developed Section 2.2, where our direct sparse convolution significantly outperforms lowering-based methods at a high level of sparsity.
Even though direct sparse convolution may seem conceptually more complicated than the usual SpMDM or the lowering-based methods, it can be concisely expressed in the pseudo code shown in Figure 2. To decouple from a specific layout of tensor I , the pseudo code uses layout function f such that f (c,y,x) maps to the offset corresponding to (c,y,x)th element of I (we assume f (c,y+r,x+s)= f (c,y,x)+ f (0,r,s)). For example, in CHW layout, f (c,y,x) = (c ·Hin + y)Win + x. In convolutional layers in CNN, an input channel is reused against multiple output channels and vice versa, and there is also ample reuse out of an input channel due to overlapping between dot-products, especially for a large filter size and a small stride. Therefore, the arithmetic intensity of sparse convolution can be significantly higher than typical sparse-matrix computations such as SpMV, thus leading to high compute efficiency. Our optimized implementation1 fully takes advantage of the reuse, applying loop tiling to both input and output channels, with column blocking (Buluç et al., 2009) applied to the weight sparse matrix. SIMDification and register blocking optimizations are also applied to the y and x loops in the pseudo code. Non-contiguous indirect access (i.e. gather) is another overhead of typical sparse-matrix computations. However, as shown in our pseudo code, the values read from colidx and value arrays of a sparse matrix are reused Hout ·Wout times. The access to tensor in is also contiguous as long as the tensor’s elements with contiguous x or y values are stored contiguously, which is a common case as in the CHW format.
2.2 ANALYTICAL PERFORMANCE MODELING: PROJECTIONS ON SPARSE CONVOLUTION SPEEDUP AND GUIDELINES ON USEFUL SPARSITY RANGE
The performance of sparse convolution depends highly on the sparsity level of the weight tensor. This section develops a performance model to determine the appropriate target sparsity range for pruning and to project theoretical speedup for any given sparsity, using the roofline model (Williams et al., 2009).
We denote the floating-point operations required in a convolution as C (in FLOP), the size of input and output activation tensors as SA (in Bytes), and the size of weight tensor as SW , all without considering sparsity. We denote the density of non-zero in filters as x (the lower the x, the higher the sparsity of weight tensor), the compute capability of processor as F (in FLOP/s), and the memory bandwidth as B (in B/s). With these parameters, the time for dense convolution (tdense), the time for sparse convolution bound by compute (tsparse_compute) and by bandwidth (tsparse_bw), and theoretical speedup can be modeled as follows (we assume dense convolution is not bandwidth bound):
tdense = C F , tsparse_compute = αxC F , tsparse_bw = SA +βxSW B , speedup= tdense max(tsparse_compute, tsparse_bw) ,
where α and β denote the compute and storage overheads of sparse representations, respectively. We observe α∼ 3 on a Xeon E5-2697 v4 processor, and β is typically 2 (in compressed sparse row representation, we need 4B column index for each 4B single-precision floating point non-zero value).
1https://github.com/IntelLabs/SkimCaffe/blob/intel_scnn/include/caffe/util/sconv.hpp
This analytical model is visualized in Figure 3. Here, we define effective FLOP/s with respect to the number of floating-point operations that would have been performed by dense convolutions including the ones for zero weights (i.e. effective FLOP/s = C/tsparse). With a moderate sparsity (e.g., x = 0.2), convolution is likely to be compute bound, and hence effective FLOP/s rapidly increases as x decreases. For conv5 in AlexNet with x ∈ (0.05,0.15), a typical sparsity range without accuracy loss, direct sparse convolution can achieve 2–7× and 4–14× speedup on Xeon and Atom platforms, respectively, as shown in Figure 3 and will be validated in Section 3.2.
However, decreasing arithmetic intensity further with lowering x eventually makes the performance bandwidth bound. Thus, there is an upper bound of useful sparsity, and a sparsity higher than it does not provide additional speedup, while only making training more challenging to preserve accuracy. This upper bound can be found by solving for x such that tsparse_compute = tsparse_bw (e.g., the upper bound sparsity for conv5 of AlexNet on the Xeon is x ∼ 0.02). This analysis can be applied to various computing platforms including CPUs and GPUs because the model captures the essential platform-dependent characteristic, the ratio of bandwidth compute capability to memory bandwidth (F/B). When the compute to bandwidth ratio is lower as in a platform like Atom, the performance will be less quickly bandwidth bound. For example, the lower bound of useful sparsity for conv5 of AlexNet is x∼ 0.01 on Atom C2750, which is smaller than that of Xeon. The speedup to sparsity relation also varies over layers. For example, since 1×1 convolutions in GoogLeNet has low arithmetic intensity to begin with, its performance quickly becomes bandwidth bound at lower sparsity (or higher x).
The compute overhead, α, depends on the quality of sparse convolution implementation and on the target processor architecture. Since tsparse_compute > tdense for x > 1/α, there is a lower bound of useful sparsity such that, with a sparsity lower than that, sparse convolution becomes slower than dense convolution. The previous section described our sparse convolution implementation that achieves α=3 (since α is the compute overhead, lower is better) on the Xeon instead of α=100 as conjectured by Szegedy et al. (2015)2.
2.3 GUIDED SPARSITY LEARNING (GSL)
The upper and lower bounds on useful sparsity can provide important insights for training/pruning. The model can tell that sparse convolution is not useful for certain layers, in which case we can skip pruning of those layers to provide more room for sparsity in the other layers. For example, layers like the first layer in AlexNet and GoogLeNet may not provide enough sparsity regardless of the amount of regularization applied as long as the original inference accuracy is to be preserved. A layer may be already bandwidth bound even before pruning like 1×1 convolution layers in GoogLeNet as shown by inception_4a/5x5_reduce layer in Figure 3.
2The compute overhead of α=3 primarily comes from that access to input tensor is not aligned at cache line boundaries. Recent Xeon processors can execute 1 unaligned SIMD load per cycle, which is not enough to sustain 2 SIMD fused multiply-add operations per cycle. In addition to this 2× overhead, when Wout is not a multiple of SIMD width (8 for Xeon E5-2697 v4), we do not fully utilize the SIMD registers. Since Atom processors do not execute multiple SIMD floating operations per cycle anyway, and because its SIMD width is narrower as 4, its compute overhead is smaller as 1.2 as will be shown in Section 3.2.
Guided Sparsity Learning (GSL), our new pruning algorithm, is inspired by the insights and our performance model. GSL is the first to fuse the awareness of speedup potential into sparsity learning. GSL is a generic algorithm and accepts different regularization methods for pruning. When GSL is used with element-wise regularization for pruning, thus denoted as Guided Element-wise Sparsity Learning (GESL), it learns the element-wise sparsity of layers where the model predicts speedups. GSL can also be used with regularization methods that are more complicated than basic ridge and lasso regularization. For example, GSL can be combined with dynamic network surgery (Guo et al., 2016), as will be shown in Section 3.1
Algorithm: Guided Sparsity Learning (GSL) Input: Pruning layer set (S), performance model (M)
Initialize Project speedup for each layer L in S using M; Exclude all Ls without speedup potential from S Repeat Train the whole neural network while actively pruning only Ls in S
Project speedup of Ls in S, using their current sparsity and M, periodically Periodically, for each L in S do:
if (sparsity ≥ upper bound of the useful sparsity range), stop pruning L if (stabilized sparsity ≤ lower bound), stop pruning L & restore its original dense weights /∗ Stop pruning L is to give other Ls better chance to prune further and achieve better accuracy ∗/
Until Maximum iterations or convergence reached
Although GSL as described above aims primarily at inference speed, GSL can balance the implications of pruning on inference speed, accuracy, and model size. To do this, optional constraints can be given to GSL to prioritize pruning of different layers in the network. For example, by using different regularization strengths on conv and fc, we can tune the priorities on speed and model size.
3 EXPERIMENTS
Our sparse CNN design is evaluated on three platforms shown in Table 1. Intel C2750 (Atom) represents resource-constrained mobile platforms or micro servers optimized for energy efficiency. Xeon E5-2697 v4 (BDW) represents data-center servers. Xeon Phi 7250 (KNL) is designed for highperformance computing, but its next version, Knights Mill, will specifically target machine learning. Our sparse CNN is implemented as an extension of Caffe deep learning framework (Jia et al., 2014) and is at https://github.com/IntelLabs/SkimCaffe. We use Intel compiler version 17.0.0 and use all cores available. The SGEMM performance and achievable memory bandwidth listed are measured with Intel MKL version 2017 and STREAM benchmark (McCalpin), respectively.
We train with the ImageNet ILSVRC-2012 dataset (Deng et al., 2009), starting from the pre-trained Caffe reference model (a slight variation but we call it AlexNet for simplicity) and GoogLeNet model from the Caffe model zoo. Since we find it is easy to have high sparsity with smaller networks and datasets like LeNet and CIFAR regardless of pruning method, we do not present their results. Our training process is based on the method described in Wen et al. (2016) with the following differences. We look for element-wise sparsity with lasso instead of group lasso, and guide the training process to target the layers and range of sparsity where we see speedup potential. We have explored various solver methods and learning rate schedules, but found that they do not significantly affect the eventual accuracy and sparsity, once hyper-parameters are tuned for the respective settings. In general, the pruning step no longer improves after 450K and 900K mini-batch iterations for AlexNet and GoogLeNet, respectively. The re-training step saturates around 150K and 300K mini-batch iterations. To see trade-offs among accuracy, speed, and model size, we try various weight decays ranging from 1e-5 to 1e-3, and, for AlexNet, decay multipliers for fc layer ranging from 1e-2 to 1. We find that the
starting learning rate of 1e-3 and weight decay of 5e-5 in general gives a high sparsity with minimal accuracy drop. We reduce the learning rate by 10× for re-training step.
3.1 GUIDED TRAINING RESULTS
Figure 4 shows the effectiveness of our guided pruning and compares the level of element-wise and group-wise sparsity we can obtain. We should look at layer-by-layer because the speedup over dense convolution does not have a simple linear relation with sparsity as shown by our model, and, therefore, the overall FLOP reduction does not necessarily closely correlate with the real speedup. In AlexNet, using the same element-wise regularization factor across all layers (element-wise sparsity learning, ESL) provides non-zero densities around 0.4 for conv2-5. This is fine sparsity when the primary goal is reducing model size, but not high enough for speeding-up inference. Therefore, guided ESL (GESL) reduces the regularization factor of fc layers (as they have fewer FLOPS) and avoid pruning conv1 entirely (as its sparsity is too low for any potential speedups with more regularization). This leads to less than 0.2 non-zero density for conv2-5, the range where we can get speedups from sparse convolution. Similarly, applying GSL to dynamic network surgery (DNS), a recent proposal to obtain high sparsity, as Guided DNS (GDNS), we can see that GSL effectively improve the obtained sparsity for accelerating inference by de-prioritizing conv1 and fc layers (we go further to not prune fc layers at all to see how much sparsity DNS can provide in conv layers)3.
Structured sparsity learning (SSL) provides group-wise sparsity, for which we can use dense methods, but its sparsity is lower because of constrained forms of sparsity. According to our model, SSL performs better when xg < (α/αg)x, where x and xg are non-zero density of ESL and SSL, and α and αg are the compute overhead of ESL and SSL, respectively. Even if we use an ideal 100% efficiency for SSL (αg = 1)4 and the measured overhead α = 3 for ESL, xg shown in Figure 4(a) is not small enough to outperform GESL. Note that our guiding principles are already applied to SSL, where conv1 and fc layers are not pruned. In short, sparsity SSL can currently obtain is too low to outperform once compared with an optimized sparse convolution design for element-wise sparsity such as ours. This motivates further investigation of pruning methods for higher group-wise sparsity.
GoogLeNet has many 1×1 convolutions, for which sparse convolution does not provide speedups due to their low arithmetic intensity, as our model predicts. As shown in Figure 4(a), GESL successfully discovers this and avoids pruning the 1×1 convolutions for higher sparsity in 3×3 and 5×5 convolutions, where our model projects speedup, and, more importantly, almost recovers the original accuracy. For 1×1 convolutions, group-wise sparsity implemented in SSL reduces to
3Although not shown in Figure 4(b), we also apply DNS and GDNS to GoogLeNet. Compared to DNS, GDNS on GoogLeNet successfully reduces non-zero density by 1.4× on average in layers with speedup potential by prioritizing these layers for pruning.
4Note that certain kinds of group-wise sparsity like “column-wise sparsity” defined in Wen et al. (2016) need lowering, which can be considerable overhead, making it hard to approach the ideal efficiency.
element-wise sparsity5 , and dense methods can no longer be used. We believe that SSL provides higher sparsity for 1×1 convolutions than ESL because SSL does not prune fc layers, providing more room to prune other layers6. For larger convolutions that contribute to the bulk of FLOPs, ESL provides significantly higher sparsity than SSL; most of the larger convolution layers have non-zero density less than 0.2, where sparse convolutions can provide speedups. It is interesting to note that ESL, GESL, and SSL all achieve very high sparsity of non-zero density less than 1.5% in layers like inception_4e/3x3. This may indicate that the 3×3 path is unnecessary in that inception module.
3.2 LAYER-BY-LAYER SPARSE CONVOLUTION RESULTS
Figure 5 shows layer-wise performance of our direct sparse convolution design with the useful high sparsity obtained from GESL. We evaluate with the sparse matrices from multiple pruned AlexNet models with up to 3% top-1 accuracy drop. Since the performance of sparse matrix operations highly depends on specific sparsity pattern, it is important not to evaluate with random sparse matrices. We use SGEMM as a proxy of dense convolution performance to quantify layer-wise speedups of direct sparse convolution. SGEMM is a good proxy because it has a long history of extensive optimizations, and it allows us not to depend on the quality of a specific dense convolution implementation7. We use batch sizes of 32, 144, and 272 for Atom, BDW, and KNL, multiples of the number of hardware threads in respective platforms.
BDW achieves 3.4× speedup with non-zero density x = 0.09, the sparsity similar to those of conv2-5 with no accuracy drop. The actual TF/s (as opposed to effective TF/s that also counts FLOPs for zeros) is 0.76 when sparse convolution is sufficiently compute bound (e.g., x > 0.4). This performance corresponds to about a third of SGEMM, from which we can derive the compute overhead of sparse convolution α as 3. As explained in Section 2.2, this leads to the lower-bound of sparsity to get speedups at x = 0.3, which matches with Figure 5(b). Atom with a higher bandwidth to flop ratio achieves higher 7.3× speedup at x = 0.09. The actual GF/s is 51 when x > 0.4, which is 1.2× lower than SGEMM performance (i.e. α = 1.2). Note that the performance projection for conv5 in Figure 3 using αs derived here resembles the measured performance in Figure 5 (conv2-5 share similar performance characteristics). KNL achieves impressive 13.9 effective TF/s at x = 0.09 (3.1× over SGEMM).
4 RELATED WORK
Recent researches have achieved great success on reducing model size and accelerating inference of CNNs while maintaining accuracy, exploring a large design space as shown in Table 2. Regularizationbased and factorization-based approaches are the two main camps. Regularization-based approaches
5This is because filter coefficients for a given input and output channel pair is also a type of group that SSL is looking for. 6However, no pruning in fc layers is not an inherent limitation of SSL. This is just because Wen et al. (2016) focus on conv layer, and we follow the same approach to see maximum sparsity that SSL and GSSL can get in conv layers. 7This is the same reason for this paper to focus on layer-wise performance instead of overall end-to-end speedup. As the baseline for overall end-to-end speedup may be relative to a baseline whose efficiency is suboptimal with performance bottlenecks in other parts/layers of the code. For more scientific comparison among different CNN speedup techniques, we recommend using dense matrix multiplication (GEMM) FLOP/s of the evaluated platform as the baseline, because many platforms readily have vendor-provided extensively-optimized GEMM implementations which can be a proxy of highly-optimized dense CNN implementation. This also aligns with a long-accepted standard practice in high performance computing community.
use a separate training step to discover and prune redundant parameters in a pre-trained model using various regularizations, including ridge (Han et al., 2015; 2016b), lasso, and group lasso (Liu et al., 2015; Wen et al., 2016), combined with thresholding. Factorization-based approaches use low-rank decomposition and can quickly produce compressed models without additional pruning steps. Both approaches can use a fine-tuning step to recover accuracy loss caused by model pruning.
Researches focusing on fully connected layers (Han et al., 2015; 2016b; Denil et al., 2013) have achieved 10–50× model size reduction for networks such as AlexNet (Krizhevsky et al., 2012). However, they achieved marginal inferencing speedup because fully connected layers usually account for less than 10% of total computation in modern CNNs. Researches in groups A and C shown in Table 2 aim at speeding up inference by focusing more on convolution layers, with most of them relying on dense methods for computing convolution. While factorization-based approaches (group C) obtain smaller models in dense format naturally, regularization-based approaches (group A) need group regularization to impose group-wise sparsity. Although Liu et al. (2015) explore sparse methods in computing convolution layers, their approach involves lowering overhead and uses hard-coding non-zeros in sparse matrix with full unrolling that leads to a large instruction footprint.
While our direct sparse convolution is demonstrated to achieve high speed up on convolution when having enough sparsity, factorization-based approaches can complement. This is because the inherent sparsity in the first few convolution layers can be not high enough, while factorization-based approaches can achieve speedups there. Liu et al. (2015) also show that factorization and regularizationbased approaches can be combined.
Winograd (Lavin & Gray, 2015) and FFT based algorithms (Vasilache et al., 2015) also aim to speedup convolution. While being orthogonal, these techniques can have synergies with our direct sparse convolution. For example, FFT based convolutions are more effective for large filters that usually reside in the first few layers where sparsity is low. While this paper focuses on convolution layer performance, our technical report (Park et al., 2016) also considers optimizations for fully connected layers, and sparsity in activations, which is also discussed in Han et al. (2016a).
5 CONCLUSIONS
Powerful CNNs are often quite compute demanding. Pruning as a post-processing step has been effective in drastically reducing the model size while boosting inference speed moderately. We aim to more fully realize the potential performance benefits due to the reduced FLOP counts resulting from pruned convolution kernels. By combining our high-performance direct sparse convolution method with a performance model, we developed a guided approach that prunes CNNs in a co-design fashion for different computer architectures and on different layers of a CNN in question. In particular, we demonstrated 3.1–7.3× convolution speedups in AlexNet on a variety of platforms, all in comparison to extensively-optimized dense linear algebra operations.
Looking ahead, as this paper shows that pruning can boost inference speed significantly in additional to reducing model size, further techniques in pruning should be explored. While our direct sparse convolution algorithm is successful, our performance model also reveals that sparse convolution cannot speedup all convolution layers, as seen from 1×1 convolutions in GoogLeNet. We plan to expand our performance model to cover other FLOP-reduction methods such as FFT, Winograd, and
tensor factorization, so that we can make informed decisions to choose the best performing method for each layer and the training process can be guided accordingly.
ACKNOWLEDGEMENT
We would like to thank Yiwen Guo, Anbang Yao, and Yurong Chen for sharing the dynamic network surgery source code and their insights. We would also like to thank Nitish Shirish Keskar for his recommendations on hyper-parameter settings.

The paper details an implementation of sparse-full convolutions and a model to work out the potential speed-up of various sparsity levels for CNNs.
The first contribution is more about engineering, but the authors make the source code available which is greatly appreciated.
The second contribution is perhaps more interesting, as so far pruning methods have focused on saving memory, with very modest speed gains. Imbuing knowledge of running speed into a pruning algorithm seems like the proper way to tackle this problem. The authors are very methodical in how they build the model and evaluate it very thoroughly.
It seems that the same idea could be used not just for pruning existing models, but also when building new architectures: selecting layers and their parameters as to achieve an optimal throughput rate. This could make for a nice direction for future work.
One point that is missing is some discussion of how transferable the performance model is to GPUs. This would make the technique easier to adopt broadly.
Other areas for improvement: The points in Figure 4 are hard to distinguish (e.g. small red circle vs. small red square), and overall the figure could be made bigger; specifying whether the "base learning rate" in Section 3 is the start or end rate of the annealing schedule; typos: "punning" (p.4), "spares" (p.5).


Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd’s algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.
1 INTRODUCTION
Deep neural networks have emerged to be the state-of-the-art in the field of machine learning for image classification, object detection, speech recognition, natural language processing, and machine translation (LeCun et al., 2015). The substantial progress of neural networks however comes with high cost of computations and hardware resources resulting from a large number of parameters. For example, Krizhevsky et al. (2012) came up with a deep convolutional neural network consisting of 61 million parameters and won the ImageNet competition in 2012. It is followed by deeper neural networks with even larger numbers of parameters, e.g., Simonyan & Zisserman (2014).
The large sizes of deep neural networks make it difficult to deploy them on resource-limited devices, e.g., mobile or portable devices, and network compression is of great interest in recent years to reduce computational cost and memory requirements for deep neural networks. Our interest in this paper is mainly on curtailing the size of the storage (memory) for network parameters (weights and biases). In particular, we focus on the network size compression by reducing the number of distinct network parameters by quantization.
Besides network quantization, network pruning has been studied for network compression to remove redundant parameters permanently from neural networks (Mozer & Smolensky, 1989; LeCun et al., 1989; Hassibi & Stork, 1993; Han et al., 2015b; Lebedev & Lempitsky, 2016; Wen et al., 2016). Matrix/tensor factorization and low-rank approximation have been investigated as well to find more efficient representations of neural networks with a smaller number of parameters and consequently to save computations (Sainath et al., 2013; Xue et al., 2013; Jaderberg et al., 2014; Lebedev et al., 2014; Yang et al., 2015; Liu et al., 2015; Kim et al., 2015; Tai et al., 2015; Novikov et al., 2015). Moreover, similar to network quantization, low-precision network implementation has been examined in Vanhoucke et al. (2011); Courbariaux et al. (2014); Anwar et al. (2015); Gupta et al. (2015); Lin et al. (2015a). Some extremes of low-precision neural networks consisting of binary or ternary parameters can be found in Courbariaux et al. (2015); Lin et al. (2015b); Rastegari et al. (2016). We note that these are different types of network compression techniques, which can be employed on top of each other.
Themost related work to our investigation in this paper can be found in Gong et al. (2014); Han et al. (2015a), where a conventional quantization method using k-means clustering is employed for network quantization. This conventional approach however is proposed with little consideration for the impact of quantization errors on the neural network performance loss and no effort to optimize the quantization procedure for a given compression ratio constraint. In this paper, we reveal the suboptimality of this conventional method and newly design quantization schemes for neural networks. In particular, we formulate an optimization problem to minimize the network performance loss due to quantization given a compression ratio constraint and find efficient quantization methods for neural networks.
The main contribution of the paper can be summarized as follows:
• It is derived that the performance loss due to quantization in neural networks can be quantified approximately by the Hessian-weighted distortion measure. Then, Hessian-weighted k-means clustering is proposed for network quantization to minimize the performance loss.
• It is identified that the optimization problem for network quantization provided a compression ratio constraint can be reduced to an entropy-constrained scalar quantization (ECSQ) problem when optimal variable-length binary coding is employed after quantization. Two efficient heuristic solutions for ECSQ are proposed for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd’s algorithm.
• As an alternative of Hessian, it is proposed to utilize some function (e.g., square root) of the second moment estimates of gradients when the Adam (Kingma & Ba, 2014) stochastic gradient decent (SGD) optimizer is used in training. The advantage of using this alternative is that it is computed while training and can be obtained at the end of training at no additional cost.
• It is shown how the proposed network quantization schemes can be applied for quantizing network parameters of all layers together at once, rather than layer-by-layer network quantization in Gong et al. (2014); Han et al. (2015a). This follows from our investigation that Hessian-weighting can handle the different impact of quantization errors properly not only within layers but also across layers. Moreover, quantizing network parameters of all layers together, one can even avoid layer-by-layer compression rate optimization.
The rest of the paper is organized as follows. In Section 2, we define the network quantization problem and review the conventional quantization method using k-means clustering. Section 3 discusses Hessian-weighted network quantization. Our entropy-constrained network quantization schemes follow in Section 4. Finally, experiment results and conclusion can be found in Section 5 and Section 6, respectively.
2 NETWORK QUANTIZATION
We consider a neural network that is already trained, pruned if employed and fine-tuned before quantization. If no network pruning is employed, all parameters in a network are subject to quantization. For pruned networks, our focus is on quantization of unpruned parameters.
The goal of network quantization is to quantize (unpruned) network parameters in order to reduce the size of the storage for them while minimizing the performance degradation due to quantization. For network quantization, network parameters are grouped into clusters. Parameters in the same cluster share their quantized value, which is the representative value (i.e., cluster center) of the cluster they belong to. After quantization, lossless binary coding follows to encode quantized parameters into binary codewords to store instead of actual parameter values. Either fixed-length binary coding or variable-length binary coding, e.g., Huffman coding, can be employed to this end.
2.1 COMPRESSION RATIO
Suppose that we have total N parameters in a neural network. Before quantization, each parameter is assumed to be of b bits. For quantization, we partition the network parameters into k clusters. Let Ci be the set of network parameters in cluster i and let bi be the number of bits of the codeword assigned to the network parameters in cluster i for 1 ≤ i ≤ k. For a lookup table to decode quantized
values from their binary encoded codewords, we store k binary codewords (bi bits for 1 ≤ i ≤ k) and corresponding quantized values (b bits for each). The compression ratio is then given by
Compression ratio = Nb
∑k i=1(|Ci|+ 1)bi + kb
. (1)
Observe in (1) that the compression ratio depends not only on the number of clusters but also on the sizes of the clusters and the lengths of the binary codewords assigned to them, in particular, when a variable-length code is used for encoding quantized values. For fixed-length codes, however, all codewords are of the same length, i.e., bi = ⌈log2 k⌉ for all 1 ≤ i ≤ k, and thus the compression ratio is reduced to only a function of the number of clusters, i.e., k, assuming thatN and b are given.
2.2 K-MEANS CLUSTERING
Provided network parameters {wi} N i=1 to quantize, k-means clustering partitions them into k disjoint sets (clusters), denoted by C1, C2, . . . , Ck, while minimizing the mean square quantization error (MSQE) as follows:
argmin C1,C2,...,Ck
k∑
i=1
∑
w∈Ci
|w − ci| 2, where ci =
1
|Ci|
∑
w∈Ci
w. (2)
We observe two issues with employing k-means clustering for network quantization.
• First, although k-means clustering minimizes the MSQE, it does not imply that k-means clustering minimizes the performance loss due to quantization as well in neural networks. K-means clustering treats quantization errors from all network parameters with equal importance. However, quantization errors from some network parameters may degrade the performance more significantly that the others. Thus, for minimizing the loss due to quantization in neural networks, one needs to take this dissimilarity into account.
• Second, k-means clustering does not consider any compression ratio constraint. It simply minimizes its distortion measure for a given number of clusters, i.e., for k clusters. This is however suboptimal when variable-length coding follows since the compression ratio depends not only on the number of clusters but also on the sizes of the clusters and assigned codeword lengths to them, which are determined by the binary coding scheme employed after clustering. Therefore, for the optimization of network quantization given a compression ratio constraint, one need to take the impact of binary coding into account, i.e., we need to solve the quantization problem under the actual compression ratio constraint imposed by the specific binary coding scheme employed after clustering.
3 HESSIAN-WEIGHTED NETWORK QUANTIZATION
In this section, we analyze the impact of quantization errors on the neural network loss function and derive that the Hessian-weighted distortion measure is a relevant objective function for network quantization in order to minimize the quantization loss locally. Moreover, from this analysis, we propose Hessian-weighted k-means clustering for network quantization to minimize the performance loss due to quantization in neural networks.
3.1 NETWORK MODEL
We consider a general non-linear neural network that yields outputy = f(x;w) from input x, where w = [w1 · · · wN ]
T is the vector consisting of all trainable network parameters in the network; N is the total number of trainable parameters in the network. A loss function loss(y, ŷ) is defined as the objective function that we aim to minimize in average, where ŷ = ŷ(x) is the expected (groundtruth) output for input x. Cross entropy or mean square error are typical examples of a loss function. Given a training data set Xtrain, we optimize network parameters by solving the following problem, e.g., approximately by using a stochastic gradient descent (SGD) method with mini-batches:
ŵ = argmin w
L(Xtrain;w), where L(X ;w) = 1
|X |
∑
x∈X
loss(f(x;w), ŷ(x)).
3.2 HESSIAN-WEIGHTED QUANTIZATION ERROR
The average loss function L(X ;w) can be expanded by Taylor series with respect tow as follows:
δL(X ;w) = g(w)T δw + 1
2 δwTH(w)δw +O(‖δw‖3), (3)
where
g(w) = ∂L(X ;w)
∂w , H(w) =
∂2L(X ;w)
∂w2 ;
the square matrix H(w) consisting of second-order partial derivatives is called as Hessian matrix or Hessian. Assume that the loss function has reached to one of its local minima, at w = ŵ, after training. At local minima, gradients are all zero, i.e., we have g(ŵ) = 0, and thus the first term in the right-hand side of (3) can be neglected at w = ŵ. The third term in the right-hand side of (3) is also ignored under the assumption that the average loss function is approximately quadratic at the local minimum w = ŵ. Finally, for simplicity, we approximate the Hessian matrix as a diagonal matrix by setting its off-diagonal terms to be zero. Then, it follows from (3) that
δL(X ; ŵ) ≈ 1
2
N∑
i=1
hii(ŵ)|δŵi| 2, (4)
where hii(ŵ) is the second-order partial derivative of the average loss function with respect to wi evaluated atw = ŵ, which is the i-th diagonal element of the Hessian matrixH(ŵ).
Now, we connect (4) with the problem of network quantization by treating δŵi as the quantization error of network parameter wi at its local optimum wi = ŵi, i.e.,
δŵi = w̄i − ŵi, (5)
where w̄i is a quantized value of ŵi. Finally, combining (4) and (5), we derive that the local impact of quantization on the average loss function at w = ŵ can be quantified approximately as follows:
δL(X ; ŵ) ≈ 1
2
N∑
i=1
hii(ŵ)|ŵi − w̄i| 2. (6)
At a local minimum, the diagonal elements of Hessian, i.e., hii(ŵ)’s, are all non-negative and thus the summation in (6) is always additive, implying that the average loss function either increases or stays the same. Therefore, the performance degradation due to quantization of a neural network can be measured approximately by the Hessian-weighted distortion as shown in (6). Further discussion on the Hessian-weighted distortion measure can be found in Appendix A.1.
3.3 HESSIAN-WEIGHTED K-MEANS CLUSTERING
For notational simplicity, we use wi ≡ ŵi and hii ≡ hii(ŵ) from now on. The optimal clustering that minimizes the Hessian-weighted distortion measure is given by
argmin C1,C2,...,Ck
k∑
j=1
∑
wi∈Cj
hii|wi − cj | 2, where cj =
∑
wi∈Cj hiiwi
∑
wi∈Cj hii
. (7)
We call this as Hessian-weighted k-means clustering. Observe in (7) that we give a larger penalty for a network parameter in defining the distortion measure for clustering when its second-order partial derivative is larger, in order to avoid a large deviation from its original value, since the impact on the loss function due to quantization is expected to be larger for that parameter.
Hessian-weighted k-means clustering is locally optimal in minimizing the quantization loss when fixed-length binary coding follows, where the compression ratio solely depends on the number of clusters as shown in Section 2.1. Similar to the conventional k-means clustering, solving this optimization is not easy, but Lloyd’s algorithm is still applicable as an efficient heuristic solution for this problem if Hessian-weighted means are used as cluster centers instead of non-weighted regular means.
3.4 HESSIAN COMPUTATION
For obtaining Hessian, one needs to evaluate the second-order partial derivative of the average loss function with respect to each of network parameters, i.e., we need to calculate
hii(ŵ) = ∂2L(X ;w)
∂w2i
∣ ∣ ∣ ∣ w=ŵ = 1 |X | ∂2 ∂w2i ∑
x∈X
loss(f(x;w), ŷ(x)) ∣ ∣ ∣ ∣ w=ŵ . (8)
Recall that we are interested in only the diagonal elements of Hessian. An efficient way of computing the diagonal of Hessian is presented in Le Cun (1987); Becker & Le Cun (1988) and it is based on the back propagation method that is similar to the back propagation algorithm used for computing first-order partial derivatives (gradients). That is, computing the diagonal of Hessian is of the same order of complexity as computing gradients.
Hessian computation and our network quantization are performed after completing network training. For the data set X used to compute Hessian in (8), we can either reuse a training data set or use some other data set, e.g., validation data set. We observed from our experiments that even using a small subset of the training or validation data set is sufficient to yield good approximation of Hessian for network quantization.
3.5 ALTERNATIVE OF HESSIAN
Although there is an efficient way to obtain the diagonal of Hessian as discussed in the previous subsection, Hessian computation is not free. In order to avoid this additional Hessian computation, we propose to use an alternative metric instead of Hessian. In particular, we consider neural networks trained with the Adam SGD optimizer (Kingma & Ba, 2014) and propose to use some function (e.g., square root) of the second moment estimates of gradients as an alternative of Hessian.
The Adam algorithm computes adaptive learning rates for individual network parameters from the first and second moment estimates of gradients. We compare the Adam method to Newton’s optimization method using Hessian and notice that the second moment estimates of gradients in the Adam method act like the Hessian in Newton’s method. This observation leads us to use some function (e.g., square root) of the second moment estimates of gradients as an alternative of Hessian.
The advantage of using the second moment estimates from the Adam method is that they are computed while training and we can obtain them at the end of training at no additional cost. It makes Hessian-weighting more feasible for deep neural networks, which have millions of parameters. We note that similar quantities can be found and used for other SGD optimization methods using adaptive learning rates, e.g., AdaGrad (Duchi et al., 2011), Adadelta (Zeiler, 2012) and RMSProp (Tieleman & Hinton, 2012).
3.6 QUANTIZATION OF ALL LAYERS
We propose quantizing the network parameters of all layers in a neural network together at once by taking Hessian-weight into account. Layer-by-layer quantization was examined in the previous work (Gong et al., 2014; Han et al., 2015a). However, e.g., in Han et al. (2015a), a larger number of bits (a larger number of clusters) are assigned to convolutional layers than fully-connected layers, which implies that they heuristically treat convolutional layers more importantly. This follows from the fact that the impact of quantization errors on the performance varies significantly across layers; some layers, e.g., convolutional layers, may be more important than the others. This concern is exactly what we can address by Hessian-weighting.
Hessian-weighting properly handles the different impact of quantization errors not only within layers but also across layers and thus it can be employed for quantizing all layers of a network together. The impact of quantization errors may vary more substantially across layers than within layers. Thus, Hessian-weighting may show more benefit in deeper neural networks. We note that Hessianweighting can still provide gain even for layer-by-layer quantization since it can address the different impact of the quantization errors of network parameters within each layer as well.
Recent neural networks are getting deeper, e.g., see Szegedy et al. (2015a;b); He et al. (2015). For such deep neural networks, quantizing network parameters of all layers together is even more advantageous since we can avoid layer-by-layer compression rate optimization. Optimizing compression
ratios jointly across all individual layers (to maximize the overall compression ratio for a network) requires exponential time complexity with respect to the number of layers. This is because the total number of possible combinations of compression ratios for individual layers increases exponentially as the number of layers increases.
4 ENTROPY-CONSTRAINED NETWORK QUANTIZATION
In this section, we investigate how to solve the network quantization problem under a constraint on the compression ratio. In designing network quantization schemes, we not only want to minimize the performance loss but also want to maximize the compression ratio. In Section 3, we explored how to quantify and minimize the loss due to quantization. In this section, we investigate how to take the compression ratio into account properly in the optimization of network quantization.
4.1 ENTROPY CODING
After quantizing network parameters by clustering, lossless data compression by variable-length binary coding can be followed for compressing quantized values. There is a set of optimal codes that achieve the minimum average codeword length for a given source. Entropy is the theoretical limit of the average codeword length per symbol that we can achieve by lossless data compression, proved by Shannon (see, e.g., Cover & Thomas (2012, Section 5.3)). It is known that optimal codes achieve this limit with some overhead less than 1 bit when only integer-length codewords are allowed. So optimal coding is also called as entropy coding. Huffman coding is one of entropy coding schemes commonly used when the source distribution is provided (see, e.g., Cover & Thomas (2012, Section 5.6)), or can be estimated.
4.2 ENTROPY-CONSTRAINED SCALAR QUANTIZATION (ECSQ)
Considering a compression ratio constraint in network quantization, we need to solve the clustering problem in (2) or (7) under the compression ratio constraint given by
Compression ratio = b
b̄+ ( ∑k i=1 bi + kb)/N > C, where b̄ =
1
N
k∑
i=1
|Ci|bi, (9)
which follows from (1). This optimization problem is too complex to solve for any arbitrary variablelength binary code since the average codeword length b̄ can be arbitrary. However, we identify that it can be simplified if optimal codes, e.g., Huffman codes, are assumed to be used. In particular, optimal coding closely achieves the lower limit of the average source code length, i.e., entropy, and then we approximately have
b̄ ≈ H = −
k∑
i=1
pi log2 pi, (10)
where H is the entropy of the quantized network parameters after clustering (i.e., source), given that pi = |Ci|/N is the ratio of the number of network parameters in cluster Ci to the number of all network parameters (i.e., source distribution). Moreover, assuming thatN ≫ k, we have
1
N
( k∑
i=1
bi + kb
)
≈ 0, (11)
in (9). From (10) and (11), the constraint in (9) can be altered to an entropy constraint given by
H = −
k∑
i=1
pi log2 pi < R,
where R ≈ b/C. In summary, assuming that optimal coding is employed after clustering, one can approximately replace a compression ratio constraint with an entropy constraint for the clustering output. The network quantization problem is then translated into a quantization problem with an entropy constraint, which is called as entropy-constrained scalar quantization (ECSQ) in information theory. Two efficient heuristic solutions for ECSQ are proposed for network quantization in the following subsections, i.e., uniform quantization and an iterative solution similar to Lloyd’s algorithm for k-means clustering.
4.3 UNIFORM QUANTIZATION
It is shown in Gish & Pierce (1968) that the uniform quantizer is asymptotically optimal in minimizing the mean square quantization error for any random source with a reasonably smooth density function as the resolution becomes infinite, i.e., as the number of clusters k → ∞. This asymptotic result leads us to come up with a very simple but efficient network quantization scheme as follows:
1. We first set uniformly spaced thresholds and divide network parameters into clusters.
2. After determining clusters, their quantized values (cluster centers) are obtained by taking the mean of network parameters in each cluster.
Note that one can use Hessian-weighted mean instead of non-weighted mean in computing cluster centers in the second step above in order to take the benefit of Hessian-weighting. A performance comparison of uniform quantization with non-weighted mean and uniform quantization with Hessian-weighted mean can be found in Appendix A.2.
Although uniform quantization is a straightforward method, it has never been shown before in the literature that it is actually one of the most efficient quantization schemes for neural networks when optimal variable-length coding, e.g., Huffman coding, follows. We note that uniform quantization is not always good; it is inefficient for fixed-length coding, which is also first shown in this paper.
4.4 ITERATIVE ALGORITHM TO SOLVE ECSQ
Another scheme proposed to solve the ECSQ problem for network quantization is an iterative algorithm, which is similar to Lloyd’s algorithm for k-means clustering. Although this iterative solution is more complicated than the uniform quantization in Section 4.3, it finds a local optimum for a given discrete source. An iterative algorithm to solve the general ECSQ problem is provided in Chou et al. (1989). We derive a similar iterative algorithm to solve the ECSQ problem for network quantization. The main difference from the method in Chou et al. (1989) is that we minimize the Hessian-weighted distortion measure instead of the non-weighted regular distortion measure for optimal quantization. The detailed algorithm and further discussion can be found in Appendix A.3.
5 EXPERIMENTS
This section presents our experiment results for the proposed network quantization schemes in three exemplary convolutional neural networks: (a) LeNet (LeCun et al., 1998) for the MNIST data set, (b) ResNet (He et al., 2015) for the CIFAR-10 data set, and (c) AlexNet (Krizhevsky et al., 2012) for the ImageNet ILSVRC-2012 data set. Our experiments can be summarized as follows:
• We employ the proposed network quantization methods to quantize all of network parameters in a network together at once, as discussed in Section 3.6.
• We evaluate the performance of the proposed network quantization methods with and without network pruning. For a pruned model, we need to store not only the values of unpruned parameters but also their respective indexes (locations) in the original model. For the index information, we compute index differences between unpruned network parameters in the original model and further compress them by Huffman coding as in Han et al. (2015a).
• For Hessian computation, 50,000 samples of the training set are reused. We also evaluate the performance when Hessian is computed with 1,000 samples only.
• Finally, we evaluate the performance of our network quantization schemes using Hessian when its alternative is used instead, as discussed in Section 3.5. To this end, we retrain the considered neural networks with the Adam SGD optimizer and obtain the second moment estimates of gradients at the end of training. Then, we use the square roots of the second moment estimates instead of Hessian and evaluate the performance.
5.1 EXPERIMENT MODELS
First, we evaluate our network quantization schemes for the MNIST data set with a simplified version of LeNet5 (LeCun et al., 1998), consisting of two convolutional layers and two fully-connected
layers followed by a soft-max layer. It has total 431,080 parameters and achieves 99.25% accuracy. For a pruned model, we prune 91% of the original network parameters and fine-tune the rest.
Second, we experiment our network quantization schemes for the CIFAR-10 data set (Krizhevsky, 2009) with a pre-trained 32-layer ResNet (He et al., 2015). The 32-layer ResNet consists of 464,154 parameters in total and achieves 92.58% accuracy. For a prunedmodel, we prune 80% of the original network parameters and fine-tune the rest.
Third, we evaluate our network quantization schemes with AlexNet (Krizhevsky et al., 2012) for the ImageNet ILSVRC-2012 data set (Russakovsky et al., 2015). We obtain a pre-trainedAlexNet Caffe model, which achieves 57.16% top-1 accuracy. For a pruned model, we prune 89% parameters and fine-tune the rest. In fine-tuning, the Adam SGD optimizer is used in order to avoid the computation of Hessian by utilizing its alternative (see Section 3.5). However, the pruned model does not recover the original accuracy after fine-tuning with the Adam method; the top-1 accuracy recovered after pruning and fine-tuning is 56.00%. We are able to find a better pruned model achieving the original accuracy by pruning and retraining iteratively (Han et al., 2015b), which is however not used here.
5.2 EXPERIMENT RESULTS
We first present the quantization results without pruning for 32-layer ResNet in Figure 1, where the accuracy of 32-layer ResNet is plotted against the average codeword length per network parameter after quantization. When fixed-length coding is employed, the proposed Hessian-weighted k-means clustering method performs the best, as expected. Observe that Hessian-weighted k-means clustering yields better accuracy than others even after fine-tuning. On the other hand, when Huffman coding is employed, uniform quantization and the iterative algorithm for ECSQ outperform Hessian-weighted k-means clustering and k-means clustering. However, these two ECSQ solutions underperformHessian-weighted k-means clustering and even k-means clustering when fixed-length coding is employed since they are optimized for optimal variable-length coding.
Figure 2 shows the performance of Hessian-weighted k-means clustering when Hessian is computed with a small number of samples (1,000 samples). Observe that even using the Hessian computed with a small number of samples yields almost the same performance. We also show the performance of Hessian-weighted k-means clustering when an alternative of Hessian is used instead of Hessian as explained in Section 3.5. In particular, the square roots of the second moment estimates of gradients are used instead of Hessian, and using this alternative provides similar performance to using Hessian.
In Table 1, we summarize the compression ratios that we can achieve with different network quantization methods for pruned models. The original network parameters are 32-bit float numbers. Using the simple uniform quantization followed by Huffman coding, we achieve the compression ratios of 51.25, 22.17 and 40.65 (i.e., the compressed model sizes are 1.95%, 4.51% and 2.46% of the original model sizes) for LeNet, 32-layer ResNet and AlexNet, respectively, at no or marginal performance loss. Observe that the loss in the compressed AlexNet is mainly due to pruning. Here, we also compare our network quantization results to the ones in Han et al. (2015a). Note that layer-bylayer quantization with k-means clustering is evaluated in Han et al. (2015a) while our quantization schemes including k-means clustering are employed to quantize network parameters of all layers together at once (see Section 3.6).
6 CONCLUSION
This paper investigates the quantization problem of network parameters in deep neural networks. We identify the suboptimality of the conventional quantization method using k-means clustering and newly design network quantization schemes so that they can minimize the performance loss due to quantization given a compression ratio constraint. In particular, we analytically show that Hessian can be used as a measure of the importance of network parameters and propose to minimize Hessianweighted quantization errors in average for clustering network parameters to quantize. Hessianweighting is beneficial in quantizing all of the network parameters together at once since it can handle the different impact of quantization errors properly not only within layers but also across layers. Furthermore, we make a connection from the network quantization problem to the entropyconstrained data compression problem in information theory and push the compression ratio to the limit that information theory provides. Two efficient heuristic solutions are presented to this end, i.e., uniform quantization and an iterative solution for ECSQ. Our experiment results show that the proposed network quantization schemes provide considerable gain over the conventional method using k-means clustering, in particular for large and deep neural networks.
A APPENDIX
A.1 FURTHER DISCUSSION ON THE HESSIAN-WEIGHTED QUANTIZATION ERROR
The diagonal approximation for Hessian simplifies the optimization problem as well as its solution for network quantization. This simplification comeswith some performance loss. We conjecture that the loss due to this approximation is small. The reason is that the contributions from off-diagonal terms are not always additive and their summationmay end up with a small value. However, diagonal terms are all non-negative and therefore their contributions are always additive. We do not verify this conjecture in this paper since solving the problem without diagonal approximation is too complex; we even need to compute the whole Hessian matrix, which is also too costly.
Observe that the relation of the Hessian-weighted distortion measure to the quantization loss holds for any model for which the objective function can be approximated as a quadratic function with respect to the parameters to quantize in the model. Hence, the quantization methods proposed in this paper to minimize the Hessian-weighted distortion measure are not specific to neural networks but are generally applicable to quantization of parameters of any model whose objective function is locally quadratic with respect to its parameters approximately.
Finally, we do not consider the interactions between quantization and retraining in our formulation in Section 3.2. We analyze the expected loss due to quantization assuming no further retraining and focus on finding optimal network quantization schemes that minimize the performance loss. In our experiments, however, we further fine-tune the quantized values (cluster centers) so that we can recover the loss due to quantization and improve the performance.
A.2 EXPERIMENT RESULTS FOR UNIFORM QUANTIZATION
We compare uniform quantization with non-weighted mean and uniform quantization with Hessianweighted mean in Figure 3, which shows that uniform quantization with Hessian-weighted mean slightly outperforms uniform quantization with non-weighted mean.
A.3 FURTHER DISCUSSION ON THE ITERATIVE ALGORITHM FOR ECSQ
In order to solve the ECSQ problem for network quantization, we define a Lagrangian cost function:
Jλ(C1, C2, . . . , Ck) = D + λH = 1
N
k∑
j=1
∑
wi∈Cj
(hii|wi − cj | 2 − λ log2 pj) ︸ ︷︷ ︸
=dλ(i,j)
, (12)
where
D = 1
N
k∑
j=1
∑
wi∈Cj
hii|wi − cj | 2, H = −
k∑
j=1
pj log2 pj .
Algorithm 1 Iterative solution for entropy-constrained network quantization
Initialization: n ← 0 Initialize the centers of k clusters: c
(0) 1 , . . . , c (0) k
Initialize the proportions of k clusters (set all of them to be the same initially): p (0) 1 , . . . , p (0) k
repeat Assignment:
for all network parameters i = 1 → N do Assign wi to the cluster j that minimizes the individual Lagrangian cost as follows:
C (n+1) l ← C (n+1) l ∪ {wi} for l = argmin
j
{
hii|wi − c (n) j | 2 − λ log2 p (n) j
}
end for Update:
for all clusters j = 1 → k do Update the cluster center and the proportion of cluster j:
c (n+1) j ←
∑
wi∈C (n+1) j hiiwi ∑
wi∈C (n+1) j
hii and p
(n+1) j ←
|C (n+1) j |
N
end for n ← n+ 1
until Lagrangian cost function Jλ decreases less than some threshold
The entropy-constrained network quantization problem is then reduced to find k partitions (clusters) C1, C2, . . . , Ck that minimize the Lagrangian cost function as follows:
argmin C1,C2,...,Ck Jλ(C1, C2, . . . , Ck).
A heuristic iterative algorithm to solve this method of Lagrange multipliers for network quantization is presented in Algorithm 1. It is similar to Lloyd’s algorithm for k-means clustering. The key difference is how to partition network parameters at the assignment step. In Lloyd’s algorithm, the Euclidean distance (quantization error) is minimized. For ECSQ, the individual Lagrangian cost function, i.e., dλ(i, j) in (12), is minimized instead, which includes both quantization error and expected codeword length after entropy coding.

This paper proposes a novel neural network compression technique.
The goal is to compress maximally the network specification via parameter quantisation with a minimum impact on the expected loss.
It assumes pruning of the network parameters has already been performed, and only considers the quantisation of the individual scalar parameters of the network.
In contrast to previous work (Han et al. 2015a, Gong et al. 2014) the proposed approach takes into account the effect of the weight quantisation on the loss function that is used to train the network, and also takes into account the effect on a variable-length binary encoding of the cluster centers used for the quantisation.
Unfortunately, the submitted paper is 20 pages, rather than the 8 recommended. The length of the paper seems unjustified to me, since the first three sections (first five pages) are very generic and redundant can be largely compressed or skipped (including figures 1 and 2). Although not a strict requirement by the submission guidelines, I would suggest the authors to compress their paper to 8 pages, this will improve the readability of the paper.
To take into account the impact on the network’s loss the authors propose to use a second order approximation of the cost function of the loss. In the case of weights that originally constitute a local minimum of the loss, this leads to a formulation of the impact of the weight quantization on the loss in terms of a weighted k-means clustering objective, where the weights are derived from the hessian of the loss function at the original weights.
The hessian can be computed efficiently using a back-propagation algorithm similar to that used to compute the gradient, as shown in cited work from the literature.
The authors also propose to alternatively use a second-order moment term used by the Adam optimisation algorithm, since it can be loosely interpreted as an approximate Hessian.
In section 4.5 the authors argue that with their approach it is more natural to quantise weights across all layers together, due to the hessian weighting which takes into account the variable impact across layers of quantisation errors on the network performance.
The last statement in this section, however, was not clear to me:
“In such deep neural networks, quantising network parameters of all layers together is more efficient since optimizing layer-by-layer clustering jointly across all layers requires exponential time complexity with respect to the number of layers.”
Perhaps the authors could elaborate a bit more on this point?
In section 5 the authors develop methods to take into account the code length of the weight quantisation in the clustering process.
The first method described by the authors (based on previous work), is uniform quantisation of the weight space, which is then further optimised by their hessian-weighted clustering procedure from section 4.
For the case of nonuniform codeword lengths to encode the cluster indices, the authors develop a modification of the Hessian weighted k-means algorithm in which the code length of each cluster is also taken into account, weighted by a factor lambda. Different values of lambda give rise to different compression-accuracy trade-offs, and the authors propose to cluster weights for a variety of lambda values and then pick the most accurate solution obtained, given a certain compression budget.
In section 6 the authors report a number of experimental results that were obtained with the proposed methods, and compare these results to those obtained by the layer-wise compression technique of Han et al 2015, and to the uncompressed models.
For these experiments the authors used three datasets, MNIST, CIFAR10 and ImageNet, with data-set specific architectures taken from the literature.
These results suggest a consistent and significant advantage of the proposed method over the work of Han et al. Comparison to the work of Gong et al 2014 is not made.
The results illustrate the advantage of the hessian weighted k-means clustering criterion, and the advantages of the variable bitrate cluster encoding.
In conclusion I would say that this is quite interesting work, although the technical novelty seems limited (but I’m not a quantisation expert).
Interestingly, the proposed techniques do not seem specific to deep conv nets, but rather generically applicable to quantisation of parameters of any model with an associated cost function for which a locally quadratic approximation can be formulated. It would be useful if the authors would discuss this point in their paper.


In Evolutionary Biology, species close in the tree of evolution are identified by similar visual features. In computer vision, deep neural networks perform image classification by learning to identify similar visual features. This leads to an interesting question: is it possible to leverage the advantage of deep networks to construct a tree of life? In this paper, we make the first attempt at building the phylogenetic tree diagram by leveraging the high-level features learned by deep neural networks. Our method is based on the intuition that if two species share similar features, then their cross activations in the softmax layer should be high. Based on the deep representation of convolutional neural networks trained for image classification, we build a tree of life for species in the image categories of ImageNet. Further, for species not in the ImageNet categories that are visually similar to some category, the cosine similarity of their activation vectors in the same layer should be high. By applying the inner product similarity of the activation vectors at the last fully connected layer for different species, we can roughly build their tree of life. Our work provides a new perspective to the deep representation and sheds light on possible novel applications of deep representation to other areas like Bioinformatics.
1 INTRODUCTION
Deep learning transforms the data into compact intermediate representations akin to principal components, and derives layered structures by removing the redundancy in representations (Li Deng, 2014). In recent years, deep learning has demonstrated great success with significant improvement in various artificial intelligence applications, including speech recognition (Sak et al., 2015), image recognition (Ciresan et al., 2012; Cir; Krizhevsky et al., 2012), and natural language processing (Vinyals et al., 2015; Socher et al., 2013).
Convolutional Neural Networks (CNNs) are mainly designed for image and video recognition. Typical CNN architecture alternates convolutional layers and pooling layers, followed by several fully connected or sparsely connected layers with a final softmax as the classification layer. Milestones include the 16-layer AlexNet (Krizhevsky et al., 2012), the 19-layer VGG (Simonyan & Zisserman, 2014), and the 22-layer GoogleNet (Szegedy et al., 2015). By adding identity function as a short cut, He et al. (2015) are able to build a substantially deeper ResNet with 152 layers, which received the first place on the ILSVRC 2015 image classification task (Russakovsky et al., 2015). Other very deep networks include the highway network with depths up to 100 layers (Srivastava et al., 2015). Eldan & Shamir (2016) provide a theoretical justification that reveals the utility of having deeper networks rather than wider networks, implying that future progress will lead to the development of even deeper networks.
Understanding the deep representations of neural networks has become increasingly difficult as the state-of-the-art models have more layers. This problem is important because it will help us understand the intrinsic mechanism of deep neural networks and explore possible novel applications based on the understanding. Ballester & de Araújo (2016) show how CNNs, trained to identify objects primarily in photos, could be used for abstract sketch recognition. Gatys et al. (2015a;b) utilize ∗The first three authors contribute equally. †Corresponding author.
the correlations between feature maps to synthesize natural textures and transfer artistic style with high perceptual quality. In Bioinformatics, deep neural networks are used for the analysis of medical images for cancer detection (Cirean et al., 2013) as well as drug discovery and toxicology (Dahl et al., 2014; Ramsundar et al., 2015; Wallach et al., 2015). A deep-learning approach based on the autoencoder architecture has been adopted to predict Gene Ontology annotations and gene-function relationships (Chicco et al., 2014).
The Tree of Life refers to the compilation of a comprehensive phylogenetic (or evolutionary) database rooted at the last universal common ancestor of life on Earth. Over the course of hundreds of millions of years, the splitting and subsequent divergence of lineages has produced the tree of life, which has as its leaves the many species of organisms (Darwin, 1859). Here we refer to a phylogenetic tree, evolutionary tree or tree of life as a branching diagram showing the inferred genealogical relationships (Evaluate how close two species are in the evolutionary history, as evaluated by observed heritable traits, such as DNA sequences) among various biological species (Hug et al., 2016). This is an important problem in evolutionary biology and many attempts have been made (Darwin, 1859; Doolittle & Bapteste, 2007; Bapteste et al., 2009; Edwards, 2009). Originally tree of life was manually built based on the understanding of the evolution history or the visual similarity of the species. Today modern techniques have been applied based on the gene similarity.
Our contributions are two-fold:
1) Provides a potential solution to the important problem of constructing a biology evolutionary tree.
We propose a novel approach in constructing a tree of life using the deep representation of CNNs trained for image classification. We conjuncture that the hierarchical feature representation learned by deep networks can be leveraged to quantify the visual similarity of the species. In this way, we might be able to construct tree of life using their feature similarity.
2) Gives insight into the representations produced by deep neural networks.
We conjecture that if images of two training categories share some similar features, then their cross activations in the softmax layer should be high. Hence we could evaluate the genetic distance of species within the training categories. Based on the deep representation of several typical CNNs, AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014) and ResNet (He et al., 2015) that are trained for ImageNet classification, we construct tree of life for dozens of species in the thousands of ImageNet categories of the training dataset.
For species not in the training categories that are visually similar to some species in the training dataset, could we still utilize their deep representation in order to judge the relationship among different species? We conjuncture that they show high cosine similarity of the activation vectors in high-level layers. By applying the inner product similarity of the activation vectors at the last fully connected layer for different species, we present empirical evidence that through transfer learning we could roughly construct their tree of life.
Experiments show that the proposed method using deep representation is very competitive to human beings in building the tree of life based on the visual similarity of the species. We also try on networks at different epochs during the training, and the quality of tree of life increases over the course of training. The performance among the three networks, AlexNet, VGG and ResNet, improves with the improvement of their classification quality.
2 THE PROPOSED METHOD
2.1 DATA COLLECTION
We have two important criterions in mind while constructing our image dataset. 1) We would like each image category, which corresponds to a node in the tree (i.e. a species), to have enough samples such that a statistic from the network activations is reasonably robust to noise. 2) There exists a ground truth hierarchy on the image categories, so we can objectively evaluate the effectiveness of our method.
Fortunately, the ImageNet 2012 Classification dataset provides the raw material we need. This dataset contains 1000 categories of common life objects, and each category contains 1000 images as the training data. Also, those categories correspond exactly to nodes in the WordNet hierarchy. WordNet (Miller, 1995) is a large lexical database of English, where words are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept and synsets are interlinked by means of conceptual-semantic and lexical relations.
For the reference network, we select three popular CNNs (AlexNet, VGG-16 and ResNet-152) trained on ImageNet. The top 5 classification errors of AlexNet, VGG and ResNet are 15.3%, 9.9% and 6.7% respectively. So they all learn the features of the images very well and we could leverage their deep representations for the ToL construction.
To find a small branch of the phylogenetic tree in order to do the reconstruction, we choose a set A of genealogically close species (species close in the evolutionary tree of life as evaluated by the branch distance) from the 1000 ImageNet categories. And for each category A ∈ A, we use all the 1000 images from the training dataset to get robust result.
For the ground truth, in the smallest WordNet subtree that contains A: 1) we could just consider the categories inA and their positions in this WordNet subtree and build a smallest ground truth tree T 1A. 2) we could additional consider some categories outsideA in this WordNet subtree. Then the ground truth tree T 2A contains some categories outside the ImageNet training categories. Note that nodes in T 1A is basically the intersection of nodes in T 2 A and nodes in the 1000 ImageNet categories. For each category outside the 1000 training categories, we also use the 1000 images from the ImageNet database 1.
2.2 SIMILARITY EVALUATION
We input all selected images for species in T 1A or T 2 A to a reference network and execute the feed forward pass. The feature maps (i.e. the activation vectors) of the last fully connected (FC) layer and the softmax layer are used to build the distance matrix.
1) The Probability Method. For T 1A, each class is in the training set and their ground truth labels are among the ones represented by the softmax layer. So we utilize the probability distribution of the images at the softmax layer in order to build a distance matrix. Specifically, for two classes of images A and B in the categories of A, we consider their cross activations in the softmax layer. For each image a ∈ A, we obtain the predicted probability Pa2B that this image belongs to node B, and we calculate the average of these values, named PA2B .
PA2B = ∑ a∈A Pa2B (1)
For each image b ∈ B, we obtain the predicted probability Pb2A that this image belongs to node A, and we calculate the average of these values, named PB2A.
PB2A = ∑ b∈B Pb2A (2)
The closer the genealogical relationship of A and B, the higher the cross predicted probability value should be. As the cross confidence is close to zero, we use the logistic function to enlarge the value. Then we add “−” to assign lower value to closer species and to keep the value nonnegative.
DAB = { 0 if A = B −log(0.5PA2B + 0.5PB2A) if A 6= B
(3)
2) The Inner Product Method. For T 2A, as some species are not in the 1000 classification categories, we use the centroid vector of the activations at the last fully connected (FC) layer for each species, and calculate the dot product of the two unitized centroid vectors to get their cosine similarity. Then we add “−” to assign lower value to closer species.
DAB = −log (
vA · vB ||vA|| ||vB ||
) (4)
1The only exception is for Bassarisk which only contains 694 images.
2.3 CONSTRUCTING THE TREE OF LIFE
Based on the distance matrix, we have three methods, namely “Approximation Central Point”, “Minimum Spanning Tree”, and “Multidimensional Scaling”, to construct a tree of life.
1) The “Approximation Central Point”(ACP) based method. In the ACP based method, we build a tree bottom up by recursively merging two species points, say A and B, with the smallest distance, and setting the distance of the new point to other points as the average distance of A and B to other points respectively.
2) The “Minimum Spanning Tree” (MST) based method. In the MST based method, we first construct a Minimum Spanning Tree (MST) based on the distance matrix. Then we build a tree from the root to the leaves, recursively split the current MST subtree into two parts by removing its longest edge until there is only one node in each subtree. In this way we build a “tree” with all the leaves corresponding to the species and closest species are splitted in the end.
3) The “Multidimensional Scaling”(MDS) based method. In the MDS based method, according to D, we know distances among the points which corresponds to the species. We first apply the MDS (Multi-Dimensional Scaling) (Borg & Groenen, 2005) algorithm to do dimension reduction and project the species points into a two dimensional subspace. Then we build a tree bottom up by recursively merging two points with the smallest Euclidean distance in the two dimensional subspace and regard the midpoint of the two merging points as the new representative point.
Our following experiments show that MST and MDS show similar performance but ACP is considerably weaker.
3 EXPERIMENTS AND ANALYSIS
We conduct a plenty set of experiments to build several branches of the phylogenetic trees of different granularity. To test whether our method could distinguish tiny visual differences, we first choose genealogically very close species, such as a set of fish species or a set of canine species, and construct their tree of life. Then, to test whether our method has good scalability for larger species, such as dog, cat, fish, etc., we choose 39 different large species to build a more general tree of life and verify whether different breeds of one large species like dogs could be grouped together. In addition, to evaluate the ability of constructing hierarchical trees based on the visual similarity of images outside the Biology, we choose some vehicle categories from the ImageNet dataset (Russakovsky et al., 2015) and build a vehicle tree.
For the methods, we use the probability method in Section 2.2 to build the distance matrix, and apply ACP, MST, and MDS based methods to build the tree of life. For the inner product method in Section 2.2, the results is slightly weaker, but it can deal with species or categories outside the training set. For details of inner product method, the readers are referred to the Appendix.
3.1 CONSTRUCTING FINE-GRAINED TREE OF LIFE
To construct fine-grained tree of life, we select several fish species of high visual similarity and test whether we could identify the tiny differences of the features. We pick six fish species from the ImageNet training set and for each species, we input all the 1000 images in the training dataset to the ResNet network.
Figure 1 shows that the tree of life constructed by MST and MDS coincides with the hierarchial tree built on WordNet. The hierarchical tree constructed by ACP does not coincide with the ground truth at all. The reason may be that in any triangle ABC, the edge length from A to the median of BC, say D, is shorter than the average length of edge AB and AC. If A is far more from symmetric as evaluated by edge BC, the recalculated distance of AD does not accurately represent the distance of A to the merged set of {B,C}. Our results demonstrate that deep CNNs could capture the local features as well as the global features simultaneously. As to rebuild tree of life for genealogically close species, we need both features of different granularity like the animal’s size, skin texture and shape. For instance, the texture of a
lionfish is very similar to that of a goldfish, then we need other features like the size to distinguish the two species.
As another example, we choose 11 very similar canine species and build a relatively lager tree, as illustrated in Figure 3. We can correctly build the canine tree, possibly according to their fur texture and shape features. The reconstructed quality is as good as what human beings could reconstruct based on the visual similarity.
3.2 CONSTRUCTING COARSE-GRAINED TREE OF LIFE
Figure 2 shows the coarse-grained tree of life for clustering species of different families by different networks: ResNet, VGG and AlexNet. We pick 38 species from five families: bird, canine, plant, fish and feline.ResNet and VGG can correctly cluster the species by families, while AlexNet has makes some mistakes. This result indicates that deep networks with higher classification quality learn the deep representations better, such that the Tree of Life built based on the deep representation also have different reconstruction quality.
To show that we not only correctly cluster the species, but also ensure the correct hierarchy within each family, we further construct a tree containing 20 species of five families, as illustrated in Figure 4.
3.3 CONSTRUCTING A VEHICLE TREE
To show the ability of building hierarchical tree for other objects other than animals, we pick eight vehicle categories from the ImageNet training set. Vehicles are very different from the animals.
Their shapes are kind of fixed and they can only do certain motions like going forward or turning around. Images of vehicles do not embed abundant features as the animal images do.
Nevertheless, our method still output good results, as shown in Figure 5. We cluster the ambulance, fire truck and garbage truck together, all of which have big carriages. While in WordNet, the ambulance is close to model T, convertible and cab, but the three do not have carriage and they are much smaller than ambulance. Our result is more reasonable than the WordNet provides.
4 CONCLUSION
By leveraging the similarity of features extracted automatically by deep learning techniques, we build a tree of life for various biological species, either belonging to the training categories or not. The results are highly competitive to the level of human beings in building the tree of life based on the visual similarity of the images. Our work provides new understandings to the deep representation of neural networks and sheds light on possible novel applications of deep learning in the area of Bioinformatics. An intriguing future work would be on how to utilize deep learning techniques to build a more delicate tree of life based on the gene similarity of the species.
ACKNOWLEDGMENTS
This research work was supported by US Army Research Office(W911NF-14-1-0477) and National Science Foundation of China(61472147).

I like this paper in that it is a creative application of computer vision to Biology. Or, at least, that would be a good narrative but I'm not confident biologists would actually care about the "Tree of Life" built from this method. There's not really any biology in this paper, either in methodology or evaluation. It boils down to a hierarchical clustering of visual categories with ground truth assumed to be the WordNet hierarchy (which may or may not be the biological ground truth inheritance relationships between species, if that is even possible to define -- it probably isn't for dog species which interbreed and it definitely isn't for vehicles) or the actual biological inheritance tree or what humans would do in the same task. If we're just worried about visual relationships and not inheritance relationships then a graph is the right structure, not a tree. A tree is needlessly lossy and imposes weird relationships (e.g. ImageNet has a photo of a "toy rabbit" and by tree distance it is maximally distant from "rabbit" because the toy is in the devices top level hierarchy and the real rabbit is in the animal branch. Are those two images really as semantically unrelated as is possible?). Our visual world is not a hierarchy. Our biological world can reasonably be defined as one. One could define the task of trying to recover the biological inheritance tree from visual inputs, although we know that would be tough to do because of situations like convergent evolution. Still, one could evaluate how well various visual features can recover the hierarchical relationship of biological organisms. This paper doesn't quite do that. And even if it did, it would still feel like a bit of a solution in search of a problem. The paper says that this type of exercise can help us understand deep features, but I'm not sure sure how much it reveals. I guess it's a fair question to ask if a particular feature produces meaningful class-to-class distances, but it's not clear that the biological tree of life or the wordnet hierarchy is the right ground truth for that (I'd argue it's not).
Finally, the paper mentions human baselines in a few places but I'm not really seeing it. "Experiments show that the proposed method using deep representation is very competitive to human beings in building the tree of life based on the visual similarity of the species." and then later "The reconstructed quality is as good as what human beings could reconstruct based on the visual similarity." That's the extent of the experiment? A qualitative result and the declaration that it's as good as humans could do?