The learning capability of a neural network improves with increasing depth at higher computational costs. Wider layers with dense kernel connectivity patterns further increase this cost and may hinder real-time inference. We propose feature map and kernel pruning for reducing the computational complexity of a deep convolutional neural network. Due to coarse nature, these pruning granularities can be exploited by GPUs and VLSI based implementations. Further, we propose a simple strategy to choose the least adversarial pruning masks. The proposed approach is generic and can select good pruning masks for feature map, kernel and intra-kernel pruning. The pruning masks are generated randomly, and the best performing one is selected using the validation set. The sufficient number of random pruning masks to try depends on the pruning ratio, and is less than 100 when 40% complexity reduction is needed. Once the least adversarial pruning mask is selected, we prune and retrain the network in one-shot. The proposed approach therefore consumes less time compared to iterative pruning. We have extensively evaluated the proposed approach with the CIFAR-100, CIFAR-10, SVHN, and MNIST datasets. Experiments show that 60-70% sparsity can be induced in the convolution layers with less than 1% increase in the misclassification rate of the baseline network.
1 INTRODUCTION
Deep and wider neural networks have the capacity to learn a complex unknown function from the training data. The network reported in Dean et al. (2012) has 1.7 billion parameters and is trained on tens of thousands of CPU cores. Similarly (Simonyan & Zisserman, 2014) has employed 11-19 layers and achieved excellent classification results on the ImageNet dataset. However, the increasing depth and width demands higher computational power. This high computational complexity is a major obstacle in porting the benefits of deep learning to resource limited devices. Further, the hotspot for optimization are the convolution layers, as most of the computations are conducted there. Therefore, many researchers have proposed ideas to accelerate deep networks for real-time inference Yu et al. (2012); Han et al. (2015b;a); Mathieu et al. (2013); Anwar et al. (2015b).
Network pruning is one promising technique that first learns a function with a sufficiently large sized network followed by removing less important connections Yu et al. (2012); Han et al. (2015b); Anwar et al. (2015b). This enables smaller networks to inherit knowledge from the large sized predecessor networks and exhibit a comparable level of performance. The works of Han et al. (2015b;a) introduce fine grained sparsity in a network by pruning scalar weights. Due to unstructured sparsity, the authors employ compressed sparse row/column (CSR/CSC) for sparse representation. Thus the fine grained irregular sparsity cannot be easily translated into computational speedups.
Sparsity in a deep convolutional neural network (CNN) can be induced at various levels. Figure 1 shows four pruning granularities. At the coarsest level, a full hidden layer can be pruned. This is shown with a red colored rectangle in Fig. 1(a). Layer wise pruning affects the depth of the network and a deep network can be converted into a shallow network. Increasing the depth improves the network performance and layer-wise pruning therefore demand intelligent techniques to mitigate the
performance degradation. The next pruning granularity is removing feature maps Polyak & Wolf (2015); Anwar et al. (2015b). Feature map pruning removes a large number of kernels and may degrade the network performance much. We therefore may not achieve higher pruning ratios with this granularity. For the depicted architecture in Fig. 1 (b)., pruning a single feature map, removes four kernels. Feature map pruning affects the layer width and we directly obtain a thinner network and no sparse representation is needed. Kernel pruning is the next pruning granularity and it prunes k × k kernels. It is neither too fine nor too coarse and is shown in Fig. 1(c). Kernel pruning is therefore a balanced choice and it can change the dense kernel connectivity pattern to a sparse one. Each convolution connection involves W × H × k × k multiply and accumulate (MAC) operations where W, H and k represents the feature map width, height and the kernel size, respectively.
Further the sparse representation for kernel pruning is also very simple. A single flag is enough to represent one convolution connection. Generally, the pruning techniques induce sparsity at the finest granularity by removing scalar weights. This sparsity can be induced in much higher rates but high pruning ratios do not directly translate into computational speedups in VLSI or parallel computer based implementations Han et al. (2015b). Figure 1(d) shows this with red colored zeroes in the kernel. Further Fig. 1 summarizes the relationship between three related factors: the pruning granularities, the pruning ratios and the sparse representations. Coarse pruning granularities demand very simple sparse representation but higher pruning ratios are comparatively difficult to achieve. Similarly fine grained pruning granularities can achieve higher pruning ratios but the sparse representation is more complicated. The proposed work therefore prunes feature maps and kernels in a network. Experimental evaluations show that better pruning results can be achieved when a network is pruned with both granularities successively.
An important contribution of this work is proposing a simple and generic strategy for the selection of pruning masks. Finding pruning candidates is an important and difficult prob-
lem. Generally, in the literature granularity specific pruning strategies are reported Han et al. (2015b); Li et al. (2016). (Anwar et al., 2015b) have developed a particle filtering approach, where the sequential importance resampling is employed. The proposed strategy randomly generates N pruning masks, evaluates the importance of each mask with the validation set, selects the best mask having the argminmi(MCRmi), prunes and retrains the network Yu et al. (2012). It is important to mention here that the pruning can be conducted in one-shot or iteratively. This difference is shown in Fig. 2. For a target pruning ratio (tpr), the iterative process gradually increases sparsity and repeats the process M times. On the other hand, the one-shot pruning induces the target pruning ratio in one step. We employ one-shot pruning as the retraining after pruning consumes much time. Thus, the one shot pruning is much more efficient in terms of the optimization time. We show experimentally that the proposed algorithm can select better pruning candidates compared to other methods. Further, our approach is not computationally expensive as it involves N random evaluations on the small sized validation set.
The rest of the paper is organized as follows. Section 2 provides detailed explanations on the pruning candidate selection. Section 3 discusses the two pruning granularities while Section 4 presents the experimental results. In Section 5, recent related works are revisited. We finally conclude the discussion in Section 6 and add the future research dimensions for this work.
2 PRUNING CANDIDATE SELECTION
Pruning reduces the number of network parameters and inevitably degrades the classification performance. The pruning candidate selection is therefore of prime importance. For a specific pruning ratio, we search for the best pruning masks which afflicts the least adversary on the pruned network. Indeed retraining can partially or fully recover the pruning losses, but the lesser the losses, the more plausible is the recovery Mishkin & Matas (2015). Further small performance degradation also means that the successor network has lost little or no knowledge of the predecessor network. If there are M potential pruning candidates, the total number of pruning masks is (2M ) and an exhaustive search is therefore infeasible even for a small sized network. We therefore propose a simple and greedy strategy for selecting pruning candidates.
We initialize a network with pre-trained parameters. These parameters may be learnt on the same or related problem. We randomly generate N pruning masks and compute the misclassification rate (MCR) for each one. We then choose the best pruning mask with maximum accuracy on the validation set. Referring to the depicted architecture in Fig.4a, suppose we need to select feature map pruning candidates in layer L2 and L3 with 1/3 pruning ratio. If N = 4, the following N ordered pairs of feature maps may be randomly selected for (L2, L3) : (1, 2), (2, 3), (3, 1), (1, 1). These combinations generate random paths in the network and we evaluate the validation set MCR through these routes in the network.
However, this further raises the question of how to approximate N. We analyze the relationship between pruning ratio and N on three datasets and the results are reported in Fig. 3. This analysis is conducted for feature map pruning but is also applicable to other pruning granularities. It can be observed from Fig. 3a and 3c, that for higher pruning ratios, bigger value of N is beneficial as it results in better pruning candidate selection. Moreover, for the pruning ratio of no more than 40%, N = 50 random evaluations generate good selections. For lower pruning ratios, retraining is also more likely to compensate the losses as the non-pruned parameters may still be in good numbers. The computational cost of this technique is not much as the evaluation is conducted on the small sized validation set. By observing Fig. 3a and 3c., we propose that the value of N can be estimated initially and later used in several pruning passes. The plots in Fig. 3b and 3d show the pre-retraining distribution of N random masks. Further, the plots in Fig. 3b and 3d, shows that the distributions are narrow for small pruning ratios.
We further analyze the effect of retraining on the pruning mask selection. We prune a network with several masks and retrain each pruned network. As several networks needs to be pruned and retrained many times, we experiment with a small network where the architecture is reported like this: 32(C5) − MP2 − 64(C5) − MP2 − 64(C5) − 64FC − 10Softmax. The network is trained with the CIFAR-10 dataset (40,000 training samples) without any data augmentation and batch normalization. The network achieves the baseline performance of 26.7% on the test set. The results are reported in Fig. 4d, where the pre and post-retraining network performance is shown on
the x and y axis, respectively. Further, we superimpose a least-squares (LS) line fit to each of the scatter plot. It can be observed that the slope of the LS line decreases for higher pruning ratios. We infer that for high pruning ratios, the final network performance is dictated by the surviving number of effective parameters. It can be observed that the overall distribution is noisy. However, in general, the pre-retraining least adversarial pruning masks perform better after retraining. In the rest of this work, we therefore use the pre-retraining best mask for pruning the network.
We further compare this method with the weight sum criterion proposed in Li et al. (2016) and shown in Fig. 4a. The set of filters or kernels from the previous layer constitute a group. This is shown with the similar color in Fig. 4a. According to Li et al. (2016), the absolute sum of weights determine the importance of a feature map. Suppose that in Fig.4a, the Layer L2 undergoes feature map pruning. The weight sum criterion computes the absolute weight sum at S1, S2 and S3. If we further suppose that the pruning ratio is 1/3, then the min(S1, S2, S3) is pruned. All the incoming and outgoing kernels from the pruned feature map are also removed. We argue that the sign of a weight in kernel plays important role in well-known feature extractors and therefore this is not a good criterion.
We compare the performance of the two algorithms and Fig. 4b and 4c shows the experimental results. These results present the network status before any retraining is conducted. We report the performance degradation in the network classification against the pruning ratio. From Fig. 4b and 4c, we can observe that our proposed method outperforms the weight sum method particularly for higher pruning ratios. The best of N Pruning masks strategy evaluates pruning candidates in combinations and provides a holistic view. The criterion in Li et al. (2016) evaluates the importance of a pruning unit in the context of a single layer while our proposed approach evaluates several paths through the network and selects the best one. The combinations work together and matter more instead of individual units. Further, our proposed technique is generic and can be used for any pruning granularity: feature map, kernel and intra-kernel pruning.
3 FEATURE MAP AND KERNEL PRUNING
In this section we discuss feature map and kernel pruning granularities. For a similar sized network, we analyze the achievable pruning ratios with feature map and kernel pruning. In terms of granularity, feature map pruning is coarser than kernel pruning. Feature map pruning does not need any sparse representation and the pruned network can be implemented in a conventional way, convolution lowering Chellapilla et al. (2006) or convolution with FFTs Mathieu et al. (2013). The proposed work analyzes the unconstrained kernel and feature map pruning. Pruning a feature map eliminates all the incoming and outgoing kernels because the outgoing kernels are no more meaningful.
Kernel pruning is comparatively finer. The dimension and connectivity pattern of 2D kernels determine the computing cost of a convolutional layer. The meshed fully connected convolution layers increases this cost and can hinder the real-time inference. In LeNet LeCun et al. (1998), the second convolution layer has 6 × 16 feature maps and the kernel connectivity has a fixed sparse pattern. With kernel pruning, we learn this pattern and convert the dense connectivity to sparse one. Kernel pruning zeroes k × k kernels and is neither too fine nor too coarse. Kernel level pruning provides a balance between fine-grained and coarse-grained pruning. It is coarser than the intra-kernel sparsity and finer than the feature map pruning. Depending on the network architecture, kernel pruning may achieve good pruning ratios at very small sparse representation and computational cost. Each convolution connection represents one convolution operation which involves width× height× k × k
MAC operations. We first select pruning candidates with the criterion outlined in Section 2. The pruned network is then retrained to compensate for the losses incurred due to pruning. Figure 5a and 5b show that depending on the network architecture, kernel pruning may achieve higher pruning ratio than feature map pruning due to finer granularity. As the sparse granularities are coarse, a generic set of computing platform can benefit from it. One disadvantage of the unconstrained kernel pruning is that the convolution unrolling technique cannot benefit from it Chellapilla et al. (2006). However, customized VLSI implementations and FFT based convolutions do not employ convolution unrolling. Mathieu et al. (2013), have proposed FFT based convolutions for faster CNN training and evaluation and the GPU based parallel implementation showed very good speedups. As commonly known that the IFFT (FFT (kernel) × FFT (featuremap)) = kernel ∗ featuremap, the kernel level pruning can relieve this task. Although the kernel size is small, massive reusability of the kernels across the mini-batch enables the use of FFT. The FFT of each kernel is computed only once and reused for multiple input vectors in a mini-batch. In a feed-forward and backward path, the summations can be carried in the FFT domain and once the sum is available, the IFFT can
Table 1: Specifications of the three CIFAR-10 networks
Network Architecture Baseline MCR(%) Data Augmentation CNNMNIST1 16(C5) − 32(C5) − 64(C5) − 120 − 10 0.62 NO CNNMNIST2 6(C5) − 16(C5) − 120(C5) − 84 − 10 0.79 NO
CNNCIFAR10.small 2 × 128C3 −MP2 − 2 × 128C3 −MP2 − 2 × 256C3 − 256FC − 10Softmax 16.6 NO CNNCIFAR10.large 2 × 128C3 −MP2 − 2 × 256C3 −MP2 − 2 × 256C3 − 1 × 512C3 − 1024FC − 1024FC − 10Softmax 9.41 YES
CNNSV HN (2 × 64C3) −MP2 − (2 × 128C3) −MP2 − (2 × 128C3) − 512FC − 512FC − 10Softmax 3.5 NO CNNCIFAR100 (2 × 128C3) −MP2 − (2 × 128C3) −MP2 − (2 × 256C3) − 256C3 − 512FC − 10Softmax 33.65 YES
Pruning Ratio 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
M C
R T
es t S
et A
ft er
R et
ra in
in g
15
15.5
16
16.5
17
17.5
18
18.5
19
19.5
20
(1). MCR Baseline = 16.260% (2). MCR Baseline + Tol(1.0) = 17.26% (3). FeatureMap Pruning (4). Feature Map Followed by Kernel Pruning (5). Kernel Pruning (6). Kernel Prune Followed by Feature Map Pruning
(a) CIFAR-10 CNNsmall
0 0.2 0.4 0.6 0.8 1 Prune Ratio
Conv2-Conv7
7
8
9
10
11
12
13
M C
R T
es t S
et A
ft er
R et
ra in
in g
FeatureMap Pruning Kernel Pruning FeatureMap followed by Kernel Pruning Baseline MCR = 9.39% Baseline + Tolerance (1.0%)
(b) CIFAR-10 CNNlarge
Figure 7: The combinations of feature map and kernel pruning is reported here. Figure (a) and (b) provides pruning results for the CNNCIFAR10.small and CNNCIFAR10.large networks. It can be observed from both figure, that more sparsity can be induced in the network by indcuing sparsity with two granularities.
be performed Mathieu et al. (2013). Similarly, a customized VLSI based implementation can also benefit from the kernel level pruning. If the VLSI implementation imposes a constraint on the pruning criterion, such as the fixed number of convolution kernels from the previous to the next layer, the pruning criterion can be adapted accordingly. In the next Section, we report and discuss the experimental results in detail. As the commonly available libraries do not support masked convolutions, we therefore profile kernel pruning with customized GPU functions. We call the GPU function only for the non-pruned convolution kernels and pass the appropriate indices. It can be observed that fewer number of convolutions will reduce the required number of GFLOPs. Howevr, we conjecture that the true benefit of kernel pruning can be obtained with FFT based masked convolution.
4 EXPERIMENTAL RESULTS
In this section, we present detailed experimental results with the CIFAR-10 and SVHN datasets Krizhevsky & Hinton (2009). We experiment on three image classification problems and induce sparsity feature map and kernel wise. We also prune one network with more than one pruning granularity in combinations. During training and pruning, we use the stochastic gradient descent (SGD) and batch normalization Ioffe & Szegedy (2015). As elaborated in Section 1, we do not prune the network in small steps, and instead one-shot prune the network for a given pruning ratio followed by retraining. The experimental results are reported in the corresponding subsections.
4.1 CIFAR-10
The CIFAR-10 dataset includes samples from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. The training set consists of 50,000 RGB samples and we allocate 20% of these samples as validation set. Test set contains 10,000 samples and each sample has 32 × 32 × RGB resolution. We evaluate the proposed pruning granularities with two networks. CNNCIFAR10.small and CNNCIFAR10.large. CNNCIFAR10.small has six convolution and two overlapped max pooling layers. We report the network architecture with an alphanumeric string as
reported in Courbariaux et al. (2015) and outlined in Table 1. The (2 × 128C3) represents two convolution layers with each having 128 feature maps and 3 × 3 convolution kernels. MP2 represents 3 × 3 overlapped max-pooling layer with a stride size of 2. We pre-process the original CIFAR-10 dataset with global contrast normalization followed by zero component analysis (ZCA) whitening.
The CNNCIFAR10.large has seven convolution and two max-pooling layers. Further, online data augmentations are employed to improve the classification accuracy. We randomly crop 28 × 28 × 3 patches from the 32×32×3 input vectors. These cropped vectors are then geometrically transformed randomly. A vector may be flipped horizontally or vertically, rotated, translated and scaled. At evaluation time, we crop patches from the four corners and the center of a 32 × 32 × 3 patch and flip it horizontally. We average the evaluation on these ten 28 × 28 × 3 patches to decide the final label. Due to larger width and depth, the CNNCIFAR10.large achieves more than 90% accuracy on the CIFAR-10 dataset. The CNNCIFAR10.small is smaller than CNNCIFAR10.large and trained without any data augmentation. The CNNCIFAR10.small therefore achieves 84% accuracy.
4.1.1 FEATURE MAP AND KERNEL LEVEL PRUNING
After layer pruning, feature map pruning is the 2nd coarsest pruning granularity. Feature map pruning reduces the width of a convolutional layer and generates a thinner network. Pruning a single feature map, zeroes all the incoming and outgoing weights and therefore, higher pruning ratios degrade the network classification performance significantly. Feature map pruning for the CNNCIFAR10.small is shown in Fig. 5a with a circle marked red colored line. The sparsity reported here is for Conv2 to Conv6. We do not pruned the first convolution layer as it has only 3 × 128 × (3 × 3) = 3456 weights. The horizontal solid line shows the baseline MCR of 16.26% whereas the dashed line shows the 1% tolerance bound. Training the network with batch normalization Ioffe & Szegedy (2015) enables us to directly prune a network for a target ratio, instead of taking small sized steps. With a baseline performance of 16.26%, the network performance is very bad at 80% feature map pruning. We can observe that 62% pruning ratio is possible with less than 1% increase in MCR. The CNNCIFAR10.small is reduced to (128C3 − 83C3)-MP3-(83C3 − 83C3)MP3-(166C3 − 166C3)-256FC-10Softmax. As pruning is only applied in Conv2 to Conv6, therefore the Figure 5a pruning ratios are computed only for these layers.
For the same network, we can see that kernel level pruning performs better. We can achieve 70% sparsity with kernel level pruning. This is attributed to the fact that kernel pruning is finer and hence it achieves higher ratios. Further kernel pruning may ultimately prune a feature map if all the incoming kernels are pruned. However at inference time, we need to define the kernel connectivity pattern which can simply be done with a binary flag. So although the sparse representation is needed, it is quite simple and straightforward. Experimental results confirm that fine grained sparsity can be induced in higher rates. We achieved 70% kernel wise sparsity for Conv2 - Conv6 and the network is compressed with very simple sparse representation.
4.1.2 COMBINATIONS OF KERNEL AND FEATURE MAP PRUNING
In this section we discuss the various pruning granularities applied in different combinations. We first apply the feature map and kernel pruning to the CNNCIFAR10.small network in different orders. With feature map pruning, we can achieve 60% sparsity under the budget of 1% increase in MCR. But at this pruning stage, the network learning capability is affected much. So we take a 50% feature map pruned network, where the CNNCIFAR10.small is reduced to (128C3 − 89C3)MP3-(89C3 − 89C3)-MP3-(179C3 − 179C3)-256FC-10Softmax. As pruning is only applied to Conv2 − Conv6, therefore in Fig. 5a., pruning ratios are computed only for these layers. This network then undergoes kernel level pruning. The blue rectangle line in Figure 7a shows the pruning
results. We achieve the best pruning results in this case and the final pruned network is reported in detail in Table 2. Overall we achieve more than 75% pruning ratio in the final pruned network.
We further conducted experiments on the CNNCIFAR10.large and the corresponding plots are shown in Fig. 7b. The CNNCIFAR10.large is much wider and deeper than the CNNsmall as reported in Table 1. Therefore there are more chances of redundancy and hence more room for pruning. Further we observe similar trends as CNNCIFAR10.small where the kernel pruning can be induced in higher ratios compared to the feature map pruning. When the kernel pruning is applied to the feature map pruned network, we can achieve more than 88% sparsity in the Conv2 −Conv7 of the CNNCIFAR10.large network. This way we show that our proposed technique has good scalability. These results are in conformity to the resiliency analysis of fixed point deep neural networks Sung et al..
4.2 CIFAR-100
The CIFAR-100 dataset has 50,000 images classified into 100 fine and 20 coarse labels. The dataset has 50,000 training and 10,000 test set images. The hundred class classification problem of CIFAR100 has 500 images for each class. We construct a validation set for learning rate scheduling during training. The validation set is constructed with 100 samples for each class from the training set. This way we are left with 400 samples per class for training. We train the network with 40,000 images with data augmentation and batch normalization Ioffe & Szegedy (2015). We obtain a baseline accuracy of 33.65% on the CIFAR-100 test set with a VGG styled network. The network architecture is reported in Table 1 as CNNCIFAR100.
The pruning plots for this dataset are provided in Fig. 8a. It can be observed that around 60% of the network parameters can be pruned with less than 1% (absolute) increase in the network performance. Moreover, pruning in combinations further improve the pruning ratios. Thus the lessons learnt generalize well to other datasets.
4.3 SVHN
The SVHN dataset consists of 32 × 32 × 3 cropped images of house numbers [Netzer et al. 2011] and bears similarity with the MNIST handwritten digit recognition dataset [LeCun et al. 1998]. The classification is challenging as more than one digit may appear in sample and the goal is to identify a digit in the center of a patch. The dataset consists of 73,257 digits for training, 26,032 for testing and 53,1131 extra for training. The extra set consists of easy samples and may augment the
training set. We generate a validation set of 6000 samples which consists of 4000 samples from the training set and 2000 samples from the extra [Sermanet et al. 2012]. The network architecture is reported like this: (2 × 64C3)-MP2- (2 × 128C3)-MP2-(2 × 128C3)-512FC-512FC-10Softmax. This network is trained with batch normalization and we achieve the baseline MCR of 3.5% on the test set. The corresponding pruning plots are reported in Fig. 8b. We can observe a similar trend where kernels can be pruned by a bigger ratio compared to feature maps. More than 70% pruning ratio can be implemented in the reported network. Thus we show that the lessons learnt generalize well on various datasets.
There can be a concern that pruning may decrease the accuracy of the original network when it is deployed in the field for run time classification. For a specific problem domain, the test set is used as a proxy for the future unseen data. We argue that to some extent, this question can be answered by comparing the per class error for the original and pruned networks. This way we can see whether the pruned network is biased towards a specific class. To analayze this, we computed the per class error with the CNNSV HN network as reported in Table 1. The results are reported in Fig. 9. It can be observed that the per class error for both validation and test set do not vary significantly. We therefore infer that the pruning and retraining process is a promising technique for complexity reduction.
5 RELATED WORK
In the literature, network pruning has been studied by several researches Han et al. (2015b;a); Yu et al. (2012); Castellano et al. (1997); Collins & Kohli (2014); Stepniewski & Keane (1997); Reed (1993). Collins & Kohli (2014) have proposed a technique where irregular sparsity is used to reduce the computational complexity in convolutional and fully connected layers. However they have not discussed how the sparse representation will affect the computational benefits. The works of Han et al. (2015b;a) introduce fine-grained sparsity in a network by pruning scalar weights. If the absolute magnitude of any weight is less than a scalar threshold, the weight is pruned. This work therefore favors learning with small valued weights and train the network with the L1/L2 norm augmented loss function. Due to pruning at very fine scales, they achieve excellent pruning ratios. However this kind of pruning results in irregular connectivity patterns and demand complex sparse representation for computational benefits. Convolutions are unrolled to matrix-matrix multiplication in Chellapilla et al. (2006) for efficient implementation. The work of Lebedev & Lempitsky (2015) also induce intra-kernel sparsity in a convolutional layer. Their target is efficient computation by unrolling convolutions as matrix-matrix multiplication. Their sparse representation is not also simple because each kernel has an equally sized pruning mask. A recently published work propose sparsity at a higher granularity and induce channel level sparsity in a CNN network for deep face application Polyak & Wolf (2015). The work of Castellano et al. (1997); Collins & Kohli (2014); Stepniewski & Keane (1997); Reed (1993) utilize unstructured fine grained sparsity in a neural network. Fixed
point optimization for deep neural networks is employed by Anwar et al. (2015a); Hwang & Sung (2014); Sung et al. for VLSI based implementations. The reference work of Anwar et al. (2015b) analyzed feature map pruning with intra-kernel strided sparsity. To reduce the size of feature map and kernel matrices, they further imposed a constraint that all the outgoing kernels from a feature map must have the same pruning mask. In this work, we do not impose any such constraint and the pruning granularities are coarser. We argue that this kind of sparsity is useful for VLSI and FFT based implementations. Moreover we show that the best pruning results are obtained when we combine feature map and kernel level pruning.
6 CONCLUDING REMARKS
In this work, we proposed feature map and kernel pruning for reducing the computational complexity of deep CNN. We have discussed that the cost of sparse representation can be avoided with coarse pruning granularities. We demonstrated a simple and generic algorithm for selecting the best pruning mask from a random pool. We showed that the proposed approach adopts a holistic approach and performs better than the other methods. Further, we adopted the efficient one-shot pruing approach as the iterative retraining consumes much time. We conducted experiments with several benchmarks and networks and showed that the proposed technique has good scalability.

This paper proposes two pruning methods to reduce the computation of deep neural network. In particular, whole feature maps and the kernel connections can be removed with not much decrease of classification accuracy.
However, this paper also has the following problems.
1)	The method is somehow trivial, since the pruning masks are mainly chosen by simple random sampling. The novelty and scalability are both limited.
2)	Experiment results are mainly focused on the classification rate and the ideal complexity. As a paper on improving computation efficiency, it should include results on practical time consumption. It is very common that reducing numbers of operations may not lead to reduced computational time on a highly parallel platform (e.g., GPU).
3)	It is more important to improve the computational efficiency on large-scale models (e.g., ImageNet classification network) than on small models (e.g., MNIST, CIFAR network). However, results on large-scale network is missing.
4)	(*Logical validity of the proposed method*) For feature map pruning, what if just to train reduced-size network is trained from scratch without transfer any knowledge from the pretrained large network? Is it possible to get the same accuracy? If so, it will simply indicate the hyper-parameter is not optimal for the original network. Experimental results are necessary to clarify the necessity of feature map pruning.
Note that I agree with that a smaller network may be more generalizable than a larger network.
----------------------------------------------
Comments to the authors's response:
Thanks for replying to my comments.
1) I still believe that the proposed methods are trivial.
2) It is nice to show GPU implementation. Compared to existing toolboxes (e.g., Torch, Caffe, TensorFlow), is the implementation of convolution efficient enough?
3) Experiments on Cifar-100 are helpful (better than cifar-10), but it is not really large-scale, where speed-up is not so critical. ImageNet and Places datasets are examples of large-scale datasets.
4) The author did not reply to the question wrt the validity of the proposed methods. This question is critical.


This paper presents a framework to tackle combinatorial optimization problems using neural networks and reinforcement learning. We focus on the traveling salesman problem (TSP) and train a recurrent neural network that, given a set of city coordinates, predicts a distribution over different city permutations. Using negative tour length as the reward signal, we optimize the parameters of the recurrent neural network using a policy gradient method. We compare learning the network parameters on a set of training graphs against learning them on individual test graphs. Without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes. Applied to the KnapSack, another NP-hard problem, the same method obtains optimal solutions for instances with up to 200 items. These results, albeit still far from state-of-the-art, give insights into how neural networks can be used as a general tool for tackling combinatorial optimization problems.
1 INTRODUCTION
Combinatorial optimization is a fundamental problem in computer science. A canonical example is the traveling salesman problem (TSP), where given a graph, one needs to search the space of permutations to find an optimal sequence of nodes with minimal total edge weights (tour length). The TSP and its variants have myriad applications in planning, manufacturing, genetics, etc. (see (Applegate et al., 2011) for an overview).
Finding the optimal TSP solution is NP-hard, even in the two-dimensional Euclidean case (Papadimitriou, 1977), where the nodes are 2D points and edge weights are Euclidean distances between pairs of points. In practice, TSP solvers rely on handcrafted heuristics that guide their search procedures to find competitive (and in many cases optimal) tours efficiently. Even though these heuristics work well on TSP, once the problem statement changes slightly, they need to be revised. In contrast, machine learning methods have the potential to be applicable across many optimization tasks by automatically discovering their own heuristics based on the training data, thus requiring less handengineering than solvers that are optimized for one task only.
While most successful machine learning techniques fall into the family of supervised learning, where a mapping from training inputs to outputs is learned, supervised learning is not applicable to most combinatorial optimization problems because one does not have access to optimal labels. However, one can compare the quality of a set of solutions using a verifier, and provide some reward feedbacks to a learning algorithm. Hence, we follow the reinforcement learning (RL) paradigm to tackle combinatorial optimization. We empirically demonstrate that, even when using optimal solutions as labeled data to optimize a supervised mapping, the generalization is rather poor compared to an RL agent that explores different tours and observes their corresponding rewards.
We propose Neural Combinatorial Optimization, a framework to tackle combinatorial optimization problems using reinforcement learning and neural networks. We consider two approaches based on policy gradients (Williams, 1992). The first approach, called RL pretraining, uses a training set to optimize a recurrent neural network (RNN) that parameterizes a stochastic policy over solutions, using the expected reward as objective. At test time, the policy is fixed, and one performs inference ∗Equal contributions. Members of the Google Brain Residency program (g.co/brainresidency).
by greedy decoding or sampling. The second approach, called active search, involves no pretraining. It starts from a random policy and iteratively optimizes the RNN parameters on a single test instance, again using the expected reward objective, while keeping track of the best solution sampled during the search. We find that combining RL pretraining and active search works best in practice.
On 2D Euclidean graphs with up to 100 nodes, Neural Combinatorial Optimization significantly outperforms the supervised learning approach to the TSP (Vinyals et al., 2015b) and obtains close to optimal results when allowed more computation time (see Figure 1). We illustrate the flexibility of the method by also applying it to the KnapSack problem, for which we get optimal results for instances with up to 200 items. Our results, while still inferior to the state-of-the-art in many dimensions (such as speed, scale and performance), give insights into how neural networks can be used as a general tool for tackling combinatorial optimization problems, especially those that are difficult to design heuristics for.
2 PREVIOUS WORK
The Traveling Salesman Problem is a well studied combinatorial optimization problem and many exact or approximate algorithms have been proposed for both Euclidean and non-Euclidean graphs. Christofides (1976) proposes a heuristic algorithm that involves computing a minimum-spanning tree and a minimum-weight perfect matching. The algorithm has polynomial running time and returns solutions that are guaranteed to be within a factor of 1.5× to optimality in the metric instance of the TSP.
The best known exact dynamic programming algorithm for TSP has a complexity of Θ(2nn2), making it infeasible to scale up to large instances, say with 40 points. Nevertheless, state of the art TSP solvers, thanks to carefully handcrafted heuristics that describe how to navigate the space of feasible solutions in an efficient manner, can solve symmetric TSP instances with thousands of nodes. Concorde (Applegate et al., 2006), widely accepted as one of the best exact TSP solvers, makes use of cutting plane algorithms (Dantzig et al., 1954; Padberg & Rinaldi, 1990; Applegate et al., 2003), iteratively solving linear programming relaxations of the TSP, in conjunction with a branch-and-bound approach that prunes parts of the search space that provably will not contain an optimal solution. Similarly, the Lin-Kernighan-Helsgaun heuristic (Helsgaun, 2000), inspired from the Lin-Kernighan heuristic (Lin & Kernighan, 1973), is a state of the art approximate search heuristic for the symmetric TSP and has been shown to solve instances with hundreds of nodes to optimality.
More generic solvers, such as Google’s vehicle routing problem solver (Google, 2016) that tackles a superset of the TSP, typically rely on a combination of local search algorithms and metaheuristics. Local search algorithms apply a specified set of local move operators on candidate solutions, based
on hand-engineered heuristics such as 2-opt (Johnson, 1990), to navigate from solution to solution in the search space. A metaheuristic is then applied to propose uphill moves and escape local optima. A popular choice of metaheuristic for the TSP and its variants is guided local search (Voudouris & Tsang, 1999), which moves out of a local minimum by penalizing particular solution features that it considers should not occur in a good solution.
The difficulty in applying existing search heuristics to newly encountered problems - or even new instances of a similar problem - is a well-known challenge that stems from the No Free Lunch theorem (Wolpert & Macready, 1997). Because all search algorithms have the same performance when averaged over all problems, one must appropriately rely on a prior over problems when selecting a search algorithm to guarantee performance. This challenge has fostered interest in raising the level of generality at which optimization systems operate (Burke et al., 2003) and is the underlying motivation behind hyper-heuristics, defined as ”search method[s] or learning mechanism[s] for selecting or generating heuristics to solve computation search problems”. Hyper-heuristics aim to be easier to use than problem specific methods by partially abstracting away the knowledge intensive process of selecting heuristics given a combinatorial problem and have been shown to successfully combine human-defined heuristics in superior ways across many tasks (see (Burke et al., 2013) for a survey). However, hyper-heuristics operate on the search space of heuristics, rather than the search space of solutions, therefore still initially relying on human created heuristics.
The application of neural networks to combinatorial optimization has a distinguished history, where the majority of research focuses on the Traveling Salesman Problem (Smith, 1999). One of the earliest proposals is the use of Hopfield networks (Hopfield & Tank, 1985) for the TSP. The authors modify the network’s energy function to make it equivalent to TSP objective and use Lagrange multipliers to penalize the violations of the problem’s constraints. A limitation of this approach is that it is sensitive to hyperparameters and parameter initialization as analyzed by (Wilson & Pawley, 1988). Overcoming this limitation is central to the subsequent work in the field, especially by (Aiyer et al., 1990; Gee, 1993). Parallel to the development of Hopfield networks is the work on using deformable template models to solve TSP. Perhaps most prominent is the invention of Elastic Nets as a means to solve TSP (Durbin, 1987), and the application of Self Organizing Map to TSP (Fort, 1988; Angeniol et al., 1988; Kohonen, 1990). Addressing the limitations of deformable template models is central to the following work in this area (Burke, 1994; Favata & Walker, 1991; Vakhutinsky & Golden, 1995). Even though these neural networks have many appealing properties, they are still limited as research work. When being carefully benchmarked, they have not yielded satisfying results compared to algorithmic methods (Sarwar & Bhatti, 2012; La Maire & Mladenov, 2012). Perhaps due to the negative results, this research direction is largely overlooked since the turn of the century.
Motivated by the recent advancements in sequence-to-sequence learning (Sutskever et al., 2014), neural networks are again the subject of study for optimization in various domains (Yutian et al., 2016), including discrete ones (Zoph & Le, 2016). In particular, the TSP is revisited in the introduction of Pointer Networks (Vinyals et al., 2015b), where a recurrent network with non-parametric softmaxes is trained in a supervised manner to predict the sequence of visited cities. Despite architecural improvements, their models were trained using supervised signals given by an approximate solver.
3 NEURAL NETWORK ARCHITECTURE FOR TSP
We focus on the 2D Euclidean TSP in this paper. Given an input graph, represented as a sequence of n cities in a two dimensional space s = {xi}ni=1 where each xi ∈ R2, we are concerned with finding a permutation of the points π, termed a tour, that visits each city once and has the minimum total length. We define the length of a tour defined by a permutation π as
L(π | s) = ∥∥xπ(n) − xπ(1)∥∥2 + n−1∑
i=1 ∥∥xπ(i) − xπ(i+1)∥∥2 , (1) where ‖·‖2 denotes `2 norm. We aim to learn the parameters of a stochastic policy p(π | s) that given an input set of points s, assigns high probabilities to short tours and low probabilities to long tours. Our neural network
architecture uses the chain rule to factorize the probability of a tour as
p(π | s) = n∏ i=1 p (π(i) | π(< i), s) , (2)
and then uses individual softmax modules to represent each term on the RHS of (2).
We are inspired by previous work (Sutskever et al., 2014) that makes use of the same factorization based on the chain rule to address sequence to sequence problems like machine translation. One can use a vanilla sequence to sequence model to address the TSP where the output vocabulary is {1, 2, . . . , n}. However, there are two major issues with this approach: (1) networks trained in this fashion cannot generalize to inputs with more than n cities. (2) one needs to have access to groundtruth output permutations to optimize the parameters with conditional log-likelihood. We address both isssues in this paper.
For generalization beyond a pre-specified graph size, we follow the approach of (Vinyals et al., 2015b), which makes use of a set of non-parameteric softmax modules, resembling the attention mechanism from (Bahdanau et al., 2015). This approach, named pointer network, allows the model to effectively point to a specific position in the input sequence rather than predicting an index value from a fixed-size vocabulary. We employ the pointer network architecture, depicted in Figure 2, as our policy model to parameterize p(π | s).
3.1 ARCHITECTURE DETAILS
Our pointer network comprises two recurrent neural network (RNN) modules, encoder and decoder, both of which consist of Long Short-Term Memory (LSTM) cells (Hochreiter & Schmidhuber, 1997). The encoder network reads the input sequence s, one city at a time, and transforms it into a sequence of latent memory states {enci}ni=1 where enci ∈ Rd. The input to the encoder network at time step i is a d-dimensional embedding of a 2D point xi, which is obtained via a linear transformation of xi shared across all input steps. The decoder network also maintains its latent memory states {deci}ni=1 where deci ∈ Rd and, at each step i, uses a pointing mechanism to produce a distribution over the next city to visit in the tour. Once the next city is selected, it is passed as the input to the next decoder step. The input of the first decoder step (denoted by 〈g〉 in Figure 2) is a d-dimensional vector treated as a trainable parameter of our neural network.
Our attention function, formally defined in Appendix A.1, takes as input a query vector q = deci ∈ Rd and a set of reference vectors ref = {enc1, . . . , enck} where enci ∈ Rd, and predicts a distribution A(ref, q) over the set of k references. This probability distribution represents the degree to which the model is pointing to reference ri upon seeing query q.
Vinyals et al. (2015a) also suggest including some additional computation steps, named glimpses, to aggregate the contributions of different parts of the input sequence, very much like (Bahdanau et al., 2015). We discuss this approach in details in Appendix A.1. In our experiments, we find that utilizing one glimpse in the pointing mechanism yields performance gains at an insignificant cost latency.
Algorithm 1 Actor-critic training 1: procedure TRAIN(training set S, number of training steps T , batch size B) 2: Initialize pointer network params θ 3: Initialize critic network params θv 4: for t = 1 to T do 5: si ∼ SAMPLEINPUT(S) for i ∈ {1, . . . , B} 6: πi ∼ SAMPLESOLUTION(pθ(.|si)) for i ∈ {1, . . . , B} 7: bi ← bθv (si) for i ∈ {1, . . . , B} 8: gθ ← 1B ∑B i=1(L(πi|si)− bi)∇θ log pθ(πi|si)
9: Lv ← 1B ∑B i=1 ‖bi − L(πi)‖ 2 2
10: θ ← ADAM(θ, gθ) 11: θv ← ADAM(θv,∇θvLv) 12: end for 13: return θ 14: end procedure
4 OPTIMIZATION WITH POLICY GRADIENTS
Vinyals et al. (2015b) proposes training a pointer network using a supervised loss function comprising conditional log-likelihood, which factors into a cross entropy objective between the network’s output probabilities and the targets provided by a TSP solver. Learning from examples in such a way is undesirable for NP-hard problems because (1) the performance of the model is tied to the quality of the supervised labels, (2) getting high-quality labeled data is expensive and may be infeasible for new problem statements, (3) one cares about finding a competitive solution more than replicating the results of another algorithm.
By contrast, we believe Reinforcement Learning (RL) provides an appropriate paradigm for training neural networks for combinatorial optimization, especially because these problems have relatively simple reward mechanisms that could be even used at test time. We hence propose to use model-free policy-based Reinforcement Learning to optimize the parameters of a pointer network denoted θ. Our training objective is the expected tour length which, given an input graph s, is defined as
J(θ | s) = Eπ∼pθ(.|s) L(π | s) . (3) During training, our graphs are drawn from a distribution S, and the total training objective involves sampling from the distribution of graphs, i.e. J(θ) = Es∼S J(θ | s) . We resort to policy gradient methods and stochastic gradient descent to optimize the parameters. The gradient of (3) is formulated using the well-known REINFORCE algorithm (Williams, 1992):
∇θJ(θ | s) = Eπ∼pθ(.|s) [( L(π | s)− b(s) ) ∇θ log pθ(π | s) ] , (4)
where b(s) denotes a baseline function that does not depend on π and estimates the expected tour length to reduce the variance of the gradients.
By drawing B i.i.d. sample graphs s1, s2, . . . , sB ∼ S and sampling a single tour per graph, i.e. πi ∼ pθ(. | si), the gradient in (4) is approximated with Monte Carlo sampling as follows:
∇θJ(θ) ≈ 1
B B∑ i=1 ( L(πi|si)− b(si) ) ∇θ log pθ(πi | si) . (5)
A simple and popular choice of the baseline b(s) is an exponential moving average of the rewards obtained by the network over time to account for the fact that the policy improves with training. While this choice of baseline proved sufficient to improve upon the Christofides algorithm, it suffers from not being able to differentiate between different input graphs. In particular, the optimal tour π∗ for a difficult graph s may be still discouraged if L(π∗|s) > b because b is shared across all instances in the batch.
Using a parametric baseline to estimate the expected tour length Eπ∼pθ(.|s)L(π | s) typically improves learning. Therefore, we introduce an auxiliary network, called a critic and parameterized
Algorithm 2 Active Search 1: procedure ACTIVESEARCH(input s, θ, number of candidates K, B, α) 2: π ← RANDOMSOLUTION() 3: Lπ ← L(π | s) 4: n← dK
B e
5: for t = 1 . . . n do 6: πi ∼ SAMPLESOLUTION(pθ(. | s)) for i ∈ {1, . . . , B} 7: j ← ARGMIN(L(π1 | s) . . . L(πB | s)) 8: Lj ← L(πj | s) 9: if Lj < Lπ then
10: π ← πj 11: Lπ ← Lj 12: end if 13: gθ ← 1B ∑B i=1(L(πi | s)− b)∇θ log pθ(πi | s) 14: θ ← ADAM(θ, gθ) 15: b← α× b+ (1− α)× ( 1
B ∑B i=1 bi)
16: end for 17: return π 18: end procedure
by θv , to learn the expected tour length found by our current policy pθ given an input sequence s. The critic is trained with stochastic gradient descent on a mean squared error objective between its predictions bθv (s) and the actual tour lengths sampled by the most recent policy. The additional objective is formulated as
L(θv) = 1
B B∑ i=1 ∥∥ bθv (si)− L(πi | si)∥∥22 . (6) Critic’s architecture for TSP. We now explain how our critic maps an input sequence s into a baseline prediction bθv (s). Our critic comprises three neural network modules: 1) an LSTM encoder, 2) an LSTM process block and 3) a 2-layer ReLU neural network decoder. Its encoder has the same architecture as that of our pointer network’s encoder and encodes an input sequence s into a sequence of latent memory states and a hidden state h. The process block, similarly to (Vinyals et al., 2015a), then performs P steps of computation over the hidden state h. Each processing step updates this hidden state by glimpsing at the memory states as described in Appendix A.1 and feeds the output of the glimpse function as input to the next processing step. At the end of the process block, the obtained hidden state is then decoded into a baseline prediction (i.e a single scalar) by two fully connected layers with respectively d and 1 unit(s).
Our training algorithm, described in Algorithm 1, is closely related to the asynchronous advantage actor-critic (A3C) proposed in (Mnih et al., 2016), as the difference between the sampled tour lengths and the critic’s predictions is an unbiased estimate of the advantage function. We perform our updates asynchronously across multiple workers, but each worker also handles a mini-batch of graphs for better gradient estimates.
4.1 SEARCH STRATEGIES
As evaluating a tour length is inexpensive, our TSP agent can easily simulate a search procedure at inference time by considering multiple candidate solutions per graph and selecting the best. This inference process resembles how solvers search over a large set of feasible solutions. In this paper, we consider two search strategies detailed below, which we refer to as sampling and active search.
Sampling. Our first approach is simply to sample multiple candidate tours from our stochastic policy pθ(.|s) and select the shortest one. In contrast to heuristic solvers, we do not enforce our model to sample different tours during the process. However, we can control the diversity of the sampled tours with a temperature hyperparameter when sampling from our non-parametric softmax (see Appendix A.2). This sampling process yields significant improvements over greedy decoding, which always selects the index with the largest probability. We also considered perturbing the pointing
mechanism with random noise and greedily decoding from the obtained modified policy, similarly to (Cho, 2016), but this proves less effective than sampling in our experiments.
Active Search. Rather than sampling with a fixed model and ignoring the reward information obtained from the sampled solutions, one can refine the parameters of the stochastic policy pθ during inference to minimize Eπ∼pθ(.|s)L(π | s) on a single test input s. This approach proves especially competitive when starting from a trained model. Remarkably, it also produces satisfying solutions when starting from an untrained model. We refer to these two approaches as RL pretraining-Active Search and Active Search because the model actively updates its parameters while searching for candidate solutions on a single test instance.
Active Search applies policy gradients similarly to Algorithm 1 but draws Monte Carlo samples over candidate solutions π1 . . . πB ∼ pθ(·|s) for a single test input. It resorts to an exponential moving average baseline, rather than a critic, as there is no need to differentiate between inputs. Our Active Search training algorithm is presented in Algorithm 2. We note that while RL training does not require supervision, it still requires training data and hence generalization depends on the training data distribution. In contrast, Active Search is distribution independent. Finally, since we encode a set of cities as a sequence, we randomly shuffle the input sequence before feeding it to our pointer network. This increases the stochasticity of the sampling procedure and leads to large improvements in Active Search.
5 EXPERIMENTS
We conduct experiments to investigate the behavior of the proposed Neural Combinatorial Optimization methods. We consider three benchmark tasks, Euclidean TSP20, 50 and 100, for which we generate a test set of 1, 000 graphs. Points are drawn uniformly at random in the unit square [0, 1]2.
5.1 EXPERIMENTAL DETAILS
Across all experiments, we use mini-batches of 128 sequences, LSTM cells with 128 hidden units, and embed the two coordinates of each point in a 128-dimensional space. We train our models with the Adam optimizer (Kingma & Ba, 2014) and use an initial learning rate of 10−3 for TSP20 and TSP50 and 10−4 for TSP100 that we decay every 5000 steps by a factor of 0.96. We initialize our parameters uniformly at random within [−0.08, 0.08] and clip the L2 norm of our gradients to 1.0. We use up to one attention glimpse. When searching, the mini-batches either consist of replications of the test sequence or its permutations. The baseline decay is set to α = 0.99 in Active Search. Our model and training code in Tensorflow (Abadi et al., 2016) will be made availabe soon. Table 1 summarizes the configurations and different search strategies used in the experiments. The variations of our method, experimental procedure and results are as follows.
Supervised Learning. In addition to the described baselines, we implement and train a pointer network with supervised learning, similarly to (Vinyals et al., 2015b). While our supervised data consists of one million optimal tours, we find that our supervised learning results are not as good as those reported in by (Vinyals et al., 2015b). We suspect that learning from optimal tours is harder for supervised pointer networks due to subtle features that the model cannot figure out only by looking at given supervised targets. We thus refer to the results in (Vinyals et al., 2015b) for TSP20 and TSP50 and report our results on TSP100, all of which are suboptimal compared to other approaches.
RL pretraining. For the RL experiments, we generate training mini-batches of inputs on the fly and update the model parameters with the Actor Critic Algorithm 1. We use a validation set of 10, 000 randomly generated instances for hyper-parameters tuning. Our critic consists of an encoder network which has the same architecture as that of the policy network, but followed by 3 processing steps and 2 fully connected layers. We find that clipping the logits to [−10, 10] with a tanh(·) activation function, as described in Appendix A.2, helps with exploration and yields marginal performance gains. The simplest search strategy using an RL pretrained model is greedy decoding, i.e. selecting the city with the largest probability at each decoding step. We also experiment with decoding greedily from a set of 16 pretrained models at inference time. For each graph, the tour found by each individual model is collected and the shortest tour is chosen. We refer to those approaches as RL pretraining-greedy and RL pretraining-greedy@16.
RL pretraining-Sampling. For each test instance, we sample 1, 280, 000 candidate solutions from a pretrained model and keep track of the shortest tour. A grid search over the temperature hyperparameter found respective temperatures of 2.0, 2.2 and 1.5 to yield the best results for TSP20, TSP50 and TSP100. We refer to the tuned temperature hyperparameter as T ∗. Since sampling does not require parameter udpates and is entirely parallelizable, we use a larger batch size for speed purposes.
RL pretraining-Active Search. For each test instance, we initialize the model parameters from a pretrained RL model and run Active Search for up to 10, 000 training steps with a batch size of 128, sampling a total of 1, 280, 000 candidate solutions. We set the learning rate to a hundredth of the initial learning rate the TSP agent was trained on (i.e. 10−5 for TSP20/TSP50 and 10−6 for TSP100).
Active Search. We allow the model to train much longer to account for the fact that it starts from scratch. For each test graph, we run Active Search for 100, 000 training steps on TSP20/TSP50 and 200, 000 training steps on TSP100.
5.2 RESULTS AND ANALYSES
We compare our methods against 3 different baselines of increasing performance and complexity: 1) Christofides, 2) the vehicle routing solver from OR-Tools (Google, 2016) and 3) optimality. Christofides solutions are obtained in polynomial time and guaranteed to be within a 1.5 ratio of optimality. OR-Tools improves over Christofides’ solutions with simple local search operators, including 2-opt (Johnson, 1990) and a version of the Lin-Kernighan heuristic (Lin & Kernighan, 1973), stopping when it reaches a local minimum. In order to escape poor local optima, ORTools’ local search can also be run in conjunction with different metaheuristics, such as simulated annealing (Kirkpatrick et al., 1983), tabu search (Glover & Laguna, 2013) or guided local search (Voudouris & Tsang, 1999). OR-Tools’ vehicle routing solver can tackle a superset of the TSP and operates at a higher level of generality than solvers that are highly specific to the TSP. While not state-of-the art for the TSP, it is a common choice for general routing problems and provides a reasonable baseline between the simplicity of the most basic local search operators and the sophistication of the strongest solvers. Optimal solutions are obtained via Concorde (Applegate et al., 2006) and LK-H’s local search (Helsgaun, 2012; 2000). While only Concorde provably solves instances to optimality, we empirically find that LK-H also achieves optimal solutions on all of our test sets after 50 trials per graph (which is the default parameter setting).
We report the average tour lengths of our approaches on TSP20, TSP50, and TSP100 in Table 2. Notably, results demonstrate that training with RL significantly improves over supervised learning
(Vinyals et al., 2015b). All our methods comfortably surpass Christofides’ heuristic, including RL pretraining-Greedy which also does not rely on search. Table 3 compares the running times of our greedy methods to the aforementioned baselines, with our methods running on a single Nvidia Tesla K80 GPU, Concorde and LK-H running on an Intel Xeon CPU E5-1650 v3 3.50GHz CPU and ORTool on an Intel Haswell CPU. We find that both greedy approaches are time-efficient but still quite far from optimality.
Searching at inference time proves crucial to get closer to optimality but comes at the expense of longer running times. Fortunately, the search from RL pretraining-Sampling and RL pretrainingActive Search can be stopped early with a small performance tradeoff in terms of the final objective. This can be seen in Table 4, where we show their performances and corresponding running times as a function of how many solutions they consider.
We also find that many of our RL pretraining methods outperform OR-Tools’ local search, including RL pretraining-Greedy@16 which runs similarly fast. Table 6 in Appendix A.3 presents the performance of the metaheuristics as they consider more solutions and the corresponding running times. In our experiments, Neural Combinatorial proves superior than Simulated Annealing but is slightly less competitive that Tabu Search and much less so than Guided Local Search.
We present a more detailed comparison of our methods in Figure 3, where we sort the ratios to optimality of our different learning configurations. RL pretraining-Sampling and RL pretrainingActive Search are the most competitive Neural Combinatorial Optimization methods and recover the optimal solution in a significant number of our test cases. We find that for small solution spaces, RL pretraining-Sampling, with a finetuned softmax temperature, outperforms RL pretraining-Active Search with the latter sometimes orienting the search towards suboptimal regions of the solution space (see TSP50 results in Table 4 and Figure 3). Furthermore, RL pretraining-Sampling benefits from being fully parallelizable and runs faster than RL pretraining-Active Search. However, for larger solution spaces, RL-pretraining Active Search proves superior both when controlling for the number of sampled solutions or the running time. Interestingly, Active Search - which starts from an untrained model - also produces competitive tours but requires a considerable amount of time (respectively 7 and 25 hours per instance of TSP50/TSP100). Finally, we show randomly picked example tours found by our methods in Figure 4 in Appendix A.4.
6 GENERALIZATION TO OTHER PROBLEMS
In this section, we discuss how to apply Neural Combinatorial Optimization to other problems than the TSP. In Neural Combinatorial Optimization, the model architecture is tied to the given combinatorial optimization problem. Examples of useful networks include the pointer network, when the output is a permutation or a truncated permutation or a subset of the input, and the classical seq2seq model for other kinds of structured outputs. For combinatorial problems that require to assign labels to elements of the input, such as graph coloring, it is also possible to combine a pointer module and a softmax module to simultaneously point and assign at decoding time. Given a model that encodes an instance of a given combinatorial optimization task and repeatedly branches into subtrees to construct a solution, the training procedures described in Section 4 can then be applied by adapting the reward function depending on the optimization problem being considered.
Additionally, one also needs to ensure the feasibility of the obtained solutions. For certain combinatorial problems, it is straightforward to know exactly which branches do not lead to any feasible solutions at decoding time. We can then simply manually assign them a zero probability when decoding, similarly to how we enforce our model to not point at the same city twice in our pointing mechanism (see Appendix A.1). However, for many combinatorial problems, coming up with a feasible solution can be a challenge in itself. Consider, for example, the Travelling Salesman Problem with Time Windows, where the travelling salesman has the additional constraint of visiting each city during a specific time window. It might be that most branches being considered early in the tour do not lead to any solution that respects all time windows. In such cases, knowing exactly which branches are feasible requires searching their subtrees, a time-consuming process that is not much easier than directly searching for the optimal solution unless using problem-specific heuristics.
Rather than explicitly constraining the model to only sample feasible solutions, one can also let the model learn to respect the problem’s constraints. A simple approach, to be verified experimentally in future work, consists in augmenting the objective function with a term that penalizes solutions for violating the problem’s constraints, similarly to penalty methods in constrained optimization. While this does not guarantee that the model consistently samples feasible solutions at inference time, this is not necessarily problematic as we can simply ignore infeasible solutions and resample from the model (for RL pretraining-Sampling and RL-pretraining Active Search). It is also conceivable to combine both approaches by assigning zero probabilities to branches that are easily identifiable as infeasible while still penalizing infeasible solutions once they are entirely constructed.
6.1 KNAPSACK EXAMPLE
As an example of the flexibility of Neural Combinatorial Optimization, we consider the KnapSack problem, another intensively studied problem in computer science. Given a set of n items i = 1...n, each with weight wi and value vi and a maximum weight capacity ofW , the 0-1 KnapSack problem consists in maximizing the sum of the values of items present in the knapsack so that the sum of the
weights is less than or equal to the knapsack capacity:
max S⊆{1,2,...,n} ∑ i∈S vi
subject to ∑ i∈S wi ≤W (7)
With wi, vi and W taking real values, the problem is NP-hard (Kellerer et al., 2004). A naive heuristic is to take the items ordered by their weight-to-value ratios until they fill up the weight capacity. Two simple heuristics are ExpKnap, which employs branch-and-bound with Linear Programming bounds (Pisinger, 1995), and MinKnap, which uses dynamic programming with enumerative bounds (Pisinger, 1997). Exact solutions can also be obtained by quantizing the weights to high precisions and then performing dynamic programming with pseudo-polynomial complexity (Bertsimas & Demir, 2002).
We apply the pointer network and encode each KnapSack instance as a sequence of 2D vectors (wi, vi). At decoding time, the pointer network points to items to include in the knapsack and stops when the total weight of the items collected so far exceeds the weight capacity. We generate three datasets, KNAP50, KNAP100 and KNAP200, of a thousand instances with items’ weights and values drawn uniformly at random in [0, 1]. Without loss of generality (since we can scale the items’ weights), we set the capacities to 12.5 for KNAP50 and 25 for KNAP100 and KNAP200. We present the performances of RL pretraining-Greedy and Active Search (which we run for 5, 000 training steps) in Table 5 and compare them to the following baselines: 1) random search (which we let sample as many feasible solutions seen by Active Search), 2) the greedy value-to-weight ratio heuristic, 3) MinKnap, 4) ExpKnap, 5) OR-Tools’ KnapSack solver (Google, 2016) and 6) optimality (which we obtained by quantizing the weights to high precisions and using dynamic programming).
7 CONCLUSION
This paper presents Neural Combinatorial Optimization, a framework to tackle combinatorial optimization with reinforcement learning and neural networks. We focus on the traveling salesman problem (TSP) and present a set of results for each variation of the framework. Experiments demonstrate that Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes. Our results, while still far from the strongest solvers (especially those which are optimized for one problem), provide an interesting research avenue for using neural networks as a general tool for tackling combinatorial optimization problems.
ACKNOWLEDGMENTS
The authors would like to thank Vincent Furnon, Mustafa Ispir, Lukasz Kaiser, Oriol Vinyals, Barret Zoph, the Google Brain team and the anonymous ICLR reviewers for insightful comments and discussion.
A APPENDIX
A.1 POINTING AND ATTENDING
Pointing mechanism: Its computations are parameterized by two attention matrices Wref ,Wq ∈ Rd×d and an attention vector v ∈ Rd as follows:
ui = { v> · tanh (Wref · ri +Wq · q) if i 6= π(j) for all j < i −∞ otherwise for i = 1, 2, ..., k (8)
A(ref, q;Wref ,Wq, v) def = softmax(u). (9)
Our pointer network, at decoder step j, then assigns the probability of visiting the next point π(j) of the tour as follows:
p(π(j)|π(< j), s) def= A(enc1:n, decj). (10)
Setting the logits of cities that already appeared in the tour to −∞, as shown in Equation 8, ensures that our model only points at cities that have yet to be visited and hence outputs valid TSP tours.
Attending mechanism: Specifically, our glimpse function G(ref, q) takes the same inputs as the attention function A and is parameterized by W gref ,W g q ∈ Rd×d and vg ∈ Rd. It performs the following computations:
p = A(ref, q;W gref ,W g q , v g) (11)
G(ref, q;W gref ,W g q , v g) def = k∑ i=1 ripi. (12)
The glimpse functionG essentially computes a linear combination of the reference vectors weighted by the attention probabilities. It can also be applied multiple times on the same reference set ref :
g0 def = q (13)
gl def = G(ref, gl−1;W g ref ,W g q , v g) (14)
Finally, the ultimate gl vector is passed to the attention function A(ref, gl;Wref ,Wq, v) to produce the probabilities of the pointing mechanism. We observed empirically that glimpsing more than once with the same parameters made the model less likely to learn and barely improved the results.
A.2 IMPROVING EXPLORATION
Softmax temperature: We modify Equation 9 as follows:
A(ref, q, T ;Wref ,Wq, v) def = softmax(u/T ), (15)
where T is a temperature hyperparameter set to T = 1 during training. When T > 1, the distribution represented by A(ref, q) becomes less steep, hence preventing the model from being overconfident.
Logit clipping: We modify Equation 9 as follows:
A(ref, q;Wref ,Wq, v) def = softmax(C tanh(u)), (16)
whereC is a hyperparameter that controls the range of the logits and hence the entropy ofA(ref, q).
A.3 OR TOOL’S METAHEURISTICS BASELINES FOR TSP
A.4 SAMPLE TOURS
(5.934)
RL pretraining -Greedy
(5.734)
RL pretraining -Sampling
(5.688)
RL pretraining -Active Search
(5.827)
Active Search
(5.688)
Optimal
RL pretraining -Greedy
RL pretraining -Sampling
RL pretraining -Active Search
Active Search
Optimal

This paper is methodologically very interesting, and just based on the methodological contribution I would vote for acceptance. However, the paper's sweeping claims of clearly beating existing baselines for TSP have been shown to not hold, with the local search method LK-H solving all the authors' instances to optimality -- in seconds on a CPU, compared to clearly suboptimal results by the authors' method in 25h on a GPU.
Seeing this clear dominance of the local search method LK-H, I find it irresponsible by the authors that they left Figure 1 as it is -- with the line for "local search" referring to an obviously poor implementation by Google rather than the LK-H local search method that everyone uses. For example, at NIPS, I saw this Figure 1 being used in a talk (I am not sure anymore by whom, but I don't think it was by the authors), the narrative being "RNNs now also clearly perform better than local search". Of course, people would use a figure like that for that purpose, and it is clearly up to the authors to avoid such misconceptions.
The right course of action upon realizing the real strength of local search with LK-H would've been to make "local search" the same line as "Optimal", showing that the authors' method is still far worse than proper local search. But the authors chose to leave the figure as it was, still suggesting that their method is far better than local search. Probably the authors didn't even think about this, but this of course will mislead the many superficial readers. To people outside of deep learning, this must look like a sensational yet obviously wrong claim. I thus vote for rejection despite the interesting method.
------------------------
Update after rebuttal and changes:
I'm torn about this paper.
On the one hand, the paper is very well written and I do think the method is very interesting and promising. I'd even like to try it and improve it in the future. So, from that point of view a clear accept.
On the other hand, the paper was using extremely poor baselines, making the authors' method appear sensationally strong in comparison, and over the course of many iterations of reviewer questions and anonymous feedback, this has come down to the authors' methods being far inferior to the state of the art. That's fine (I expected that all along), but the problem is that the authors don't seem to want this to be true... E.g., they make statements, such as "We find that both greedy approaches are time-efficient and just a few percents worse than optimality."
That statement may be true, but it is very well known in the TSP community that it is typically quite trivial to get to a few percent worse than optimality. What's hard and interesting is to push those last few percent.
(As a side note: the authors probably don't stop LK-H once it has found the optimal solution, like they do with their own method after finding a local optimum. LK-H is an anytime algorithm, so even if it ran for a day that doesn't mean that it didn't find the optimal solution after milliseconds -- and a solution a few percent suboptimal even faster).
Nevertheless, since the claims have been toned down over the course of the many iterations, I was starting to feel more positive about this paper when just re-reading it. That is, until I got to the section on Knapsack solving. The version of the paper I reviewed was not bad here, as it at least stated two simple heuristics that yield optimal solutions:
"Two simple heuristics are ExpKnap, which employs brand-and-bound with Linear Programming bounds (Pisinger, 1995), and MinKnap, which employs dynamic programming with enumerative bounds (Pisinger, 1997). Exact solutions can also be optained by quantizing the weights to high precisions and then performing dynamic programming with a pseudo-polynomial complexity (Bertsimas & Demir, 2002)." That version then went on to show that these simple heuristics were already optimal, just like their own method.
In a revision between December 11 and 14, however, that paragraph, along with the optimal results of ExpKnap and MinKnap seems to have been dropped, and the authors instead introduced two new poor baseline methods (random search and greedy). This was likely in an effort to find some methods that are not optimal on these very easy instances. I personally find it pointless to present results for random search here, as nobody would use that for TSP. It's like comparing results on MNIST against a decision stump (yes, you'll do better than that, but that is not surprising). The results for greedy are interesting to see. However, dropping the strong results of the simple heuristics ExpKnap and MinKnap (and their entire discussion) appears unresponsible, since the resulting table in the new version of the paper now suggests that the authors' method is better than all baselines. Of course, if all that one is after is a column of bold numbers for ones own approach that's what one can do, but I don't find it responsible to hide the better baselines. Also, why don't the authors try at least the same OR-tools solver from Google that they tried for TSP? It seems to support Knapsack directly: