Previous work has shown that feature maps of deep convolutional neural networks (CNNs) can be interpreted as feature representation of a particular image region. Features aggregated from these feature maps have been exploited for image retrieval tasks and achieved state-of-the-art performances in recent years. The key to the success of such methods is the feature representation. However, the different factors that impact the effectiveness of features are still not explored thoroughly. There are much less discussion about the best combination of them. The main contribution of our paper is the thorough evaluations of the various factors that affect the discriminative ability of the features extracted from CNNs. Based on the evaluation results, we also identify the best choices for different factors and propose a new multi-scale image feature representation method to encode the image effectively. Finally, we show that the proposed method generalises well and outperforms the state-of-the-art methods on four typical datasets used for visual instance retrieval.
Previous work has shown that feature maps of deep convolutional neural networks (CNNs) can be interpreted as feature representation of a particular image region. Features aggregated from these feature maps have been exploited for image retrieval tasks and achieved state-of-the-art performances in recent years. The key to the success of such methods is the feature representation. However, the different factors that impact the effectiveness of features are still not explored thoroughly. There are much less discussion about the best combination of them. The main contribution of our paper is the thorough evaluations of the various factors that affect the discriminative ability of the features extracted from CNNs. Based on the evaluation results, we also identify the best choices for different factors and propose a new multi-scale image feature representation method to encode the image effectively. Finally, we show that the proposed method generalises well and outperforms the state-of-the-art methods on four typical datasets used for visual instance retrieval.
1 INTRODUCTION
Image retrieval is an important problem both for academic research and for industrial applications. Although it has been studied for many years (Sivic & Zisserman, 2003; Philbin et al., 2007; Tolias et al., 2015), it is still a challenging task. Generally, image retrieval is divided into two groups. The first one is the category-level image retrieval (Sharma & Schiele, 2015), in which an image in the dataset is deemed to be similar to the query image if they share the same class or they are similar in shape and local structures. The other group is the instance-level image retrieval (Tolias et al., 2015), in which an image is considered to match the query if they contain the same object or the same scene. The instance-level image retrieval is harder in that the retrieval method need to encode the local and detailed information in order to tell two images apart, e.g., the algorithm should be able to detect the differences between the Eiffel Tower and other steel towers although they have similar shapes. In this paper, we focus on the instance-level image retrieval.
Traditionally, visual instance retrieval is mainly addressed by the BoF (bag of features) based methods using the local feature descriptors such as SIFT (Lowe, 2004). In order to boost the retrieval performances, post-processing techniques such as query expansion (Chum et al., 2007) and spatial verification (Philbin et al., 2007) are also employed.
With the decisive victory (Krizhevsky et al., 2012) over traditional models in the ImageNet (Russakovsky et al., 2015) image classification challenge, convolutional neural networks (Lecun et al., 1998) continue to achieve remarkable success in diverse fields such as object detection (Liu et al., 2015; Shaoqing Ren, 2015), semantic segmentation (Dai et al., 2016) and even image style transfer (Gatys et al., 2016). Networks trained on the Imagenet classification task can generalize quite well to other tasks, which are either used off-the-shelf (Razavian et al., 2014a) or fine-tuned on the task-specific datasets (Azizpour et al., 2014; Long et al., 2015). Inspired by all these, researchers in the field of image retrieval also shift their interest to the CNNs. Their experiments have shown promising and surprising results (Babenko et al., 2014; Razavian et al., 2014c; Tolias et al., 2015), which are on par with or surpass the performances of conventional methods like BoF and VLAD (vector of locally aggregated descriptors) (Jégou et al., 2010; Arandjelović & Zisserman, 2013) .
Despite all these previous advances (Babenko et al., 2014; Babenko & Lempitsky, 2015; Tolias et al., 2015) on using CNNs for image feature representation, the underlying factors that contribute to the success of off-the-shelf CNNs on the image retrieval tasks are still largely unclear and unexplored, e.g., which layer is the best choice for instance retrieval, the convolutional layer or the fully-connected layer? What is the best way to represent the multi-scale information of an image? Clarifying these questions will help us advance a further step towards building a more robust and accurate retrieval system. Also in situations where a large numbers of training samples are not available, instance retrieval using unsupervised method is still preferable and may be the only option.
In this paper, we aim to answer these questions and make three novel contributions. Unlike previous papers, we explicitly choose five factors to study the image representations based on CNNs and conduct extensive experiments to evaluate their impacts on the retrieval performances. We also give detailed analysis on these factors and give our recommendations for combining them. During experiments, we borrow wisdoms from literatures and evaluate their usefulness, but find that they are not as effective as some of the simpler design choices. Second, by combining the insights obtained during the individual experiments, we are able to propose a new multi-scale image representation, which is compact yet effective. Finally, we evaluate our method on four challenging datasets, i.e., Oxford5k, Paris6k, Oxford105k and UKB. Experimental results show that our method is generally applicable and outperforms all previous methods on compact image representations by a large margin.
2 RELATED WORK
Multi-scale image representation. Lazebnik et al. (2006) propose the spatial pyramid matching approach to encode the spatial information using BoF based methods. They represent an image using a pyramid of several levels or scales. Features from different scales are combined to form the image representation in such a way that coarser levels get less weight while finer levels get more weight. Their argument is that matches found in coarser levels may involve increasingly dissimilar image features. In our paper, we also explore the multi-scale paradigm in the same spirit using the convolutional feature maps as the local descriptors. We find that the deep features from the convolutional feature maps are distinct from the traditional descriptors: the weighted sum of different level of features shows no superior performances than a simple summation of them. Kaiming et al. (2014) devise an approach called SPP (spatial pyramid pooling). In SPP, feature maps of the last convolutional layer are divided into a 3 or 4 scale pyramid. First the regional features in each scale are concatenated, then the scale-level features are concatenated to a fixed length vector to be forwarded to the next fully-connected layers. We find that this strategy is ineffective for unsupervised instance retrieval, leading to inferior performances compared to other simple combination methods (see the part about multi-scale representation in section 5.2 for more details.).
Image representation using off-the-shelf CNNs. Gong et al. (2014) propose the MOP (multiscale orderless pooling) method to represent an image in which VLAD is used to encode the level 2 and level 3 features. Then features from different scales are PCA-compressed and concatenated to form the image features. This method is rather complicated and time-consuming. At the same time, Babenko et al. (2014) use Alexnet (Krizhevsky et al., 2012) trained on the Imagenet 1000-class classification task and retrain the network on task-related dataset. The retraining procedure gives a boost to the retrieval performances. Instead of using the output of the fully-connected layers as the image feature representations, Babenko & Lempitsky (2015) use the output feature maps of last convolutional layer to compute the image features. Recently, instead of sum-pooling the convolutional features, Tolias et al. (2015) use max-pooling to aggregate the deep descriptors. Their multi-scale method, called R-MAC (regional maximum activation of convolutions), further improves the previous results on four common instance retrieval datasets. Our work differs from these papers in that we explicitly explore the various factors that underpin the success of unsupervised instance retrieval, which have not been fully explored and analysed. By carefully choosing the different setting for each factor and combining them in a complementary way, we show that a large improvement can be achieved without additional cost.
3 IMPACTING FACTORS
When we employ off-the-shelf CNNs for the task of instance-level image retrieval, a natural question is: what kind of design choices should we make in order to make full use of the representational power of existing models? In this section, we summarize the five factors that may greatly impact the performance of the final image retrieval system. In section 5.2, we will show our experimental results on each key factor. Before we delve into the impacting factors, first we will give a brief introduction about how to represent an image using the activation feature maps of a certain layer.
3.1 CNN FEATURES FOR INSTANCE RETRIEVAL
In this paper, we are mainly interested in extracting compact and discriminative image features using the off-the-shelf CNNs in an efficient way. For a given image I , we simply subtract the mean value of the RGB channels from the original image and do not do other sophisticated preprocessing. Then the image is fed into the convolutional network and goes through a series of convolutions, non-linear activations and pooling operations. The feature activation maps of a certain layer can be interpreted as the raw image features, based on which we build the final image features. These feature maps form a tensor of size K ×H ×W , where K is the number of feature channels, and H and W are height and width of a feature map. Each feature map represents a specific pattern which encodes a small part of information about the original image. If we represent the set of feature maps as F = {Fi}, i = 1, 2, . . . ,K, where Fi is the ith activation feature map, then the most simple image feature is formulated as:
f = [f1, f2, . . . , fi, . . . , fK ] T . (1)
In the above equation 1, fi is obtained by applying the feature aggregation method (see section 3.2) over the ith feature map Fi. Throughout this paper, we use feature maps after the non-linear activations (ReLU) so that the elements in each feature map are all non-negative. We also experiment with feature maps prior to ReLU, but find that they lead to inferior performances. After the image feature representation is obtained, post-processing techniques such as PCA and whitening can be further applied.
3.2 IMPACTING FACTORS ON PERFORMANCE
Feature aggregation and normalization. After the feature maps of a certain layer are obtained, it is still challenging to aggregate the 3-dimensional feature maps to get compact vector representations for images. Previous papers use either sum-pooling (Babenko & Lempitsky, 2015) or maxpooling (Tolias et al., 2015) followed by l2-normalization. Sum-pooling over a particular feature map Fi is expressed as
fi = H∑ m=1 W∑ n=1 Fi(m,n), i ∈ {1, 2, . . . ,K}, (2)
while max-pooling is given by fi = max
m,n Fi(m,n), (3)
where m,n are all the possible values over the spatial coordinate of size H × W . In this paper, for the first time, different combinations of aggregation and normalization methods (l2 and l1 in the manner of RootSIFT (Arandjelović & Zisserman, 2012)) are evaluated and their results are reported.
Output layer selection. Zeiler & Fergus (2014) has shown that image features aggregated from the feature activation maps of certain layers have interpretable semantic meanings. Gong et al. (2014) and Babenko et al. (2014) use the output of the first fully-connected layer to obtain the image features, while Babenko & Lempitsky (2015) and Tolias et al. (2015) use the output feature maps of the last convolutional layer. But these choices are somewhat subjective. In this paper, we extract dataset image features from the output feature maps of different layers and compare their retrieval performances. Based on the finding in this experiment, we choose the best-performing layer and also come up with a layer ensemble approach which outperforms state-of-the-art methods (see section 5.3).
Image resizing. Famous models such as Alexnet (Krizhevsky et al., 2012) and VGGnet (Simonyan & Zisserman, 2014) all require that the input images have fixed size. In order to meet this requirement, previous papers (Gong et al., 2014; Babenko & Lempitsky, 2015) usually resize the input
images to the fixed size. We postulate that the resizing operation may lead to the distortion of important information about the objects in the natural images. Ultimately, this kind of operation may hurt the discriminative power of image features extracted from the network, thus degrading the retrieval performances. For the task of image retrieval, we think it is best to keep the images their original sizes and feed them directly to the network whenever possible. In this paper, three image resizing strategies are explored:
• Both the height and width of the dataset images are set to the same fixed value (denoted as two-fixed).
• The minimum of each dataset image’s size is set to a fixed value. (The aspect ratio of the original image is kept.) (denoted as one-fixed).
• The images are kept their original sizes. (denoted as free).
Multi-scale feature representation. Unlike local feature descriptors such as SIFT (Lowe, 2004), the feature vector extracted from the deep convolutional networks for an image is a global descriptor which encodes the holistic information. When used for image retrieval, this kind of features still lack the detailed and local information desired to accurately match two images. Inspired by spatial pyramid matching (Lazebnik et al., 2006) and SPP (Kaiming et al., 2014), we explore the feasibility of applying this powerful method to obtain discriminative image features. An image is represented by a L-level pyramid, and at each level, the image is divided evenly into several overlapping or non-overlapping regions. The vector representations of these small regions are computed, then the regional vectors are combined to form the image feature vectors. The single scale representation of an image is just a special case of the multi-scale method in which the number of level L equals 1.
Figure 1 shows an example of 3 level representations of an image. The time cost of re-feeding those small regions into the network to compute the regional vectors would be huge, thus unacceptable for instance retrieval tasks. Inspired by the work of Girshick (2015) and Tolias et al. (2015), we assume a linear projection between the original image regions and the regions in the feature maps of a certain layer. Then the regional feature vectors can be efficiently computed without re-feeding the corresponding image regions. In section 5.2, various settings for the multi-scale and scalelevel feature combination methods are explored and their retrieval performances are reported and analysed.
PCA and whitening. Principal Component Analysis (PCA) is a simple yet efficient method for reducing the dimensionality of feature vectors and decorrelating the feature elements. Previous work (Babenko et al., 2014; Jégou et al., 2010) has shown evidences that PCA and whitened features can actually boost the performances of image retrieval. In this paper, we further investigate the usefulness of PCA and whitening within our pipeline and give some recommendations.
4 IMPLEMENTATION
We use the open source deep learning framework Caffe (Jia et al., 2014) for our whole experiments. The aim of this research is to investigate the most effective ways to exploit the feature activations of existing deep convolutional models. Based on past practices for networks to go deeper (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2015), a consideration for moderate computational cost, and also the results from Tolias et al. (2015) that deeper networks work better than shallower ones, we decide to use the popular VGG-19 model (Simonyan & Zisserman, 2014) trained on ImageNet as our model.
Network transformation. The original VGG-19 network only accepts an image of fixed size (224× 224), which is not the optimal choice when extracting image features for retrieval tasks. In order for the network to be able to process an image of arbitrary size (of course, the image size can not exceed the GPU’s memory limit) and for us to experiment with different input image resizing strategies, we adapt the original VGG-19 network and change the fully-connected layers to convolutional (Long et al., 2015) layers. For more details about network transformations, see appendix A.
5 EXPERIMENTS
In this section, we first introduce the datasets used and the evaluation metrics. Then we report our experimental results for different impacting factors and give detailed analysis. In the last part, we show the performance of our method considering all these impacting factors and compare our method with the state-of-the-art methods on four datasets.
5.1 DATASETS AND EVALUATION METRICS
The Oxford5k dataset (Philbin et al., 2007) contains 5062 images crawled from Flickr by using 11 Oxford landmarks as queries. A total of 11 groups of queries — each having 5 queries with their ground truth relevant image list, are provided. For each query, a bounding box annotation is also provided to denote the query region. During experiment, we report results using the full query images (denoted as full-query) and image regions within the bounding boxes of the query images (denoted as cropped-query). The performance on this dataset is measured by mAP (mean average precision) over all queries.
The Paris6k dataset (Philbin et al., 2008) includes 6412 images1 from Flickr which contains 11 landmark buildings and the general scenes from Paris. Similar to the Oxford5k dataset, a total of 55 queries belonging to 11 groups and the ground truth bounding boxes for each query are provided . The performance is reported as mAP over 55 queries.
The Oxford105k2 dataset contains the original Oxford5k dataset and additional 100,000 images (Philbin et al., 2007) from Flickr. The 100,000 images are disjoint with the Oxford5k dataset and are used as distractors to test the retrieval performance when the dataset scales to larger size. We use the same evaluation protocol as the Oxford5k on this dataset.
The UKB dataset (Nistér & Stewénius, 2006) consists of 10200 photographs of 2550 objects, each object having exactly 4 images. The pictures of these objects are all taken indoor with large variation in orientation, scale, lighting and shooting angles. During experiment, each image is used to query the whole dataset. The performance is measured by the average number of same-object images in the top-4 results.
5.2 RESULTS AND DISCUSSION
In this section, we report the results of experiments on the impact of different factors and analyse their particular impact. The experiments in this section are conducted on the Oxford5k dataset.
Feature aggregation and normalization. In this experiment, we compare the different combinations of feature aggregation (sum-pooling and max-pooling) and normalization methods (l2 and l1)
1 Following conventions, 20 corrupted images from this dataset are removed, leaving 6392 valid images. 2The image named “portrait 000801.jpg” was corrupted and manually removed from this dataset.
in terms of their retrieval performances. We use features from the layer conv5 4 with the free input image size. The results (%) are shown in Table 1. Sum-pooling followed by l1 normalization leads to slightly better results than the other combinations, especially for the cropped-query. However, after preliminary experiment with a multi-scale version of sum-l1 and max-l2, we find that max-l2 is much better than sum-l1. For example, employing a 4 level representation of images in the Oxford5k dataset, for the case of full-query, we find that the mAP for the max-l2 method is 65.1, while the mAP for sum-l1 is only 51.3 (even lower than the single scale representation). Base on these results, we stick to max-l2 in computing the final image features.
Output layer selection. In order to verify their feasibility for instance retrieval, we extract from the network the output feature maps of different layers and aggregate them to get the image feature vectors. We evaluate the performances using features from layer conv3 3 up to the highest fc7-conv layer (except the pooling layers, i.e. pool3, pool4 and pool5). Single-scale representations of the dataset images are used in this experiment.
Figure 2 shows the retrieval performances of image features corresponding to different layers. The retrieval performances for both the full and cropped queries increase as the layer increases from lower layer conv3 3 to higher layers and plateau in layer conv5 4 and fc6-conv, then the performances begin to decrease as the layers increase to fc7-conv. The result shows that features from lower layers such as conv3 3 and conv3 4 are too generic and lack the semantic meanings of the object in the image, thus rendering them unsuitable for instance retrieval. On the other hand, features from the highest layer (fc7-conv) contain the semantic meaning of objects but lack the detailed and local information needed to match two similar images. The best results are obtained in layer conv5 4 (0.601) and fc6-conv (0.618), where the feature vectors combine both the low-level detailed information and high level semantic meanings of the image. Based on these observations and the requirement for keeping the image features compact, we mainly focus on image features from the layer conv5 4 (dimensionality = 512 compared to 4096 of layer fc6-conv).
Image resizing. We experiment with 3 kinds of image resizing strategies which are detailed in section 3.2. We use grid search to find the optimal size for the two-fixed and one-fixed strategy. As is shown in Table 2, the free input strategy outperforms or is close to the other two strategies: it
performs especially well in the cropped-query case. This experiment shows that changing the image aspect ratio (two-fixed) distorts the image information, thus reducing the performance dramatically. The one-fixed way is better than the two-fixed method. But information loss still occurs due to the resizing operation. The free method is able to capture more natural and un-distorted information from the images, which explains its superior performance over the other two methods. It is best to keep the images their original sizes for the instance retrieval tasks.
The benefit of multi-scale representation. In our multi-scale approach, the regional vectors from each scale are simply added together and l2-normalized to form the scale-level feature vectors. Then features from different scales are combined and l2-normalized to form the image representations. In fact, we also experimented with two methods which concatenate features from different scales. The first method is in same vein to spatial pyramid pooling (Kaiming et al., 2014), i.e., region-level as well as the scale-level features are all concatenated to form a high dimensional vector. In the second method, region-level features are added while scale-level features are concatenated. We find that these two methods all lead to inferior results. The performance drop for the first in the case of cropped-query can be as large as 41%. The high dimensionality of the concatenated features (larger than 1.5k) will also lead to longer running times. Considering all these, we do not use concatenation of features in the following experiments.
We conduct extensive experiments to decide the best configurations for the multi-scale approach and report our results in Table 3. First, we explore the impact of the number of scales on the retrieval performances. For the 2 and 3 scale representations, The region number for each level are {1 × 1, 2×2 }, {1×1, 2×2, 3×3}. For the 4 scale representation, 3 versions are used and they differ in the number of regions in each scale: for “v1”, “v2”, and “v3”, the number of regions are {1× 1, 2× 2, 3×3, 4×4}, {1×1, 2×2, 3×3, 5×5} and {1×1, 2×2, 3×3, 6×6}. Table 3 (a1)(b1)(c6) show the performances of using 2, 3, and 4 scales to represent the dataset images, respectively. Clearly, more scale levels improve the results and in the case of cropped-query, increase the performance by an absolute 2%.
We also conduct experiments to find whether the weighing of different scales leads to improved performance. The weighing method for features from different scales is similar to the manner of spatial pyramid matching (Lazebnik et al., 2006) — features from coarser level are given less weight while features from the finer levels are given more weight. Suppose the features of different scales for an L scale representation are f1, f2, . . . , fL, then the image representation f is expressed as:
f = 1
2L−1 f1 + L∑ i=2
1
2L−i+1 f i. (4)
More details can be found in Lazebnik et al. (2006). Comparing the results of row (a1) and (a2), it seems that weighing different scales leads to better performance. But after more experiments, we find that the weighing method generally leads to inferior results as the number of scales increase,
16 80 144 208 272 336 400 464 528
e.g., compare the results of row pair(b1)(b2) and (c1)(c2). These results suggest that deep features are different from the traditional local feature descriptors such as SIFT. We should exercise with caution when we apply the traditional wisdom found in SIFT to the deep convolutional descriptors, which is also suggested in Babenko & Lempitsky (2015). Based on the results of this experiment, no weighing methods are used in computing our final image feature representations.
Next, we look into the issue of overlapping between different scales and try to verify its usefulness. For each scale and its different versions, we set some overlapping areas between the neighboring regions in either one or two scales of the pyramid (For the exact configurations of overlap in all cases in Table 3, see appendix B for the complete descriptions). From the row pair (b1)(b3) and (c1)(c3), we can see that overlap increase the performance for full-query but decrease a little the performance for cropped-query. But for 4 scale v3 (note the pair(c7)(c8)), we see a consistent improvement for both the full and cropped queries. So we decided to use overlap in level 2 and 3 in computing our final features.
PCA and whitening. We perform PCA and whitening for the features extracted from the Oxford5k dataset using the PCA and whitening matrix learned from the Oxford5k or the Paris6k dataset and l2-normalize these features to get the final image representations.
The retrieval results for 3 groups of features (from Table 3(b3)(c1)(c8)) are shown in Table 4. Clearly, PCA and whitening lead to better performances. For all 3 groups of features, PCA and
whitening on the same dataset lead to insignificant improvement both in the case of full and cropped query. But after doing PCA and whitening on the Paris6k dataset, the results for both the full and cropped queries improve greatly. In fact, the improvement for the case of cropped-query is even more surprising. For example, for the third feature group, the improvement are 10.4% and 13.4% for the full and cropped queries. It should also be noted that as the the number of principal component reserved increases, the performance for “PCA on self” and “PCA on Paris” differs greatly. As is shown in Figure 3, the performance for the former peaks at a relatively low dimension (around 100) and begins to decrease, while for the latter, the performance increases as the number of principal component gets larger and then plateaus.
Do the above results mean that we should always compute the PCA and whitening matrix from any datasets other than the query dataset itself? The short answer is no. We find that for UKB, learning the PCA and whitening matrix on the Oxford5k dataset shows inferior results compared to learning the PCA and whitening matrix on UKB itself (about 2% drop in accuracy). This may be due to the large differences between the images of the two datasets as the Oxford5k dataset are mainly images of buildings while the images in UKB are mainly small indoor objects. We therefore recommend learning the PCA and whitening matrix on a similar dataset to achieve good performances.
5.3 COMPARISON WITH OTHER METHODS
Based on the previous experimental results and our analysis of different impacting factors on the retrieval performances, we propose a new multi-scale image feature representation. For a given image in the dataset, the whole process of image feature representation is divided into two steps. First, the input image is fed into the network without the resizing operation (the free way) and a 4-scale feature representation is built on top of the feature maps of layer conv5 4. During the multiscale representation step, max-pooling of feature maps are used and regional vectors from the same scale are added together and l2-normalized. After that, features from different scales are summed and l2-normalized again. The second step involves applying the PCA and whitening operations on features from the first step. The PCA and whitening matrix used are either learned from different or same dataset: specifically, for the Oxford5k and Oxford105k, it is learned in the Paris6k, while for Paris6k and UKB, it is learned on Oxford5k and UKB respectively. The final PCA and whitened image features are used for reporting our method’s performances.
Layer ensemble. Inspired by previous work on model ensemble to boost the classification performances (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), we consider fusing the similarity score from different layers to improve the retrieval performances. Specifically, for two images, their similarity score is computed as the weighted sum of the scores from different layers (these weights sum to 1 so that overall similarity score between two images are still in the range [0, 1].). We have evaluated various combination of layers to see their performances and find that best performance is achieved by combining the score from conv5 4 and fc6-conv. For the fc6-conv features of an image, we use a 3-scale representation as the size of output feature maps are already very small.
The fc6-conv features are compressed to low dimensional vectors for faster computation. Our layer ensemble achieves 75.6% and 73.7% on Oxford5k for the full and cropped queries respectively, showing a large improvement over previous methods. This suggests that features from the fc6-conv and conv5 4 are complementary. See Table 5 for the complete results on all four datasets.
Comparison. We compare the performance of our method with several state-of-the-art methods which use small footprint representations and do not employ the complicated post-processing techniques such as geometric re-ranking (Philbin et al., 2007) and query expansion (Arandjelović & Zisserman, 2012). The results are shown in Table 5. In all the datasets and different scenarios (full or cropped), our method achieves the best performance with comparable cost. For Oxford5k (cropped) and UKB dataset, the relative improvement of our best results over previous methods (from Tolias et al. (2015) and Babenko & Lempitsky (2015)) are 10.3% and 4.4%.
6 CONCLUSION
In this paper, we focus on instance retrieval based on features extracted from CNNs. we have conducted extensive experiments to evaluate the impact of five factors on the performances of image retrieval and analysed their particular impacts. Based on the insights gained from these experiments, we have proposed a new multi-scale image representation which shows superior performances over previous methods on four datasets. When combined with the technique “layer ensemble”, our method can achieve further improvements. Overall, we have provided a viable and efficient solution to apply CNNs in an unsupervised way to datasets with a relatively small number of images.
APPENDIX A THE NETWORK TRANSFORMATIONS
In order for the network to process images of varying sizes, We change the layer fc6, fc7 and fc8 from the original model to fc6-conv, fc7-conv and fc8-conv. It should be noted there are certain constraints on the input image size due to the network’s inherent design. The original network accepts an image of fixed size (224×224), so the output feature maps of the last convolutional layer conv5 4 is of size 512 × 7 × 7. As a result, when we change the operation between layer conv5 4 and fc6 from inner product to convolution, each filter bank kernel between conv5 4 and fc6-conv has size 7 × 7. This in turn means that if we are to extract features from layer fc6-conv and above, the minimum size of an input image must equal to or be greater than 224. For output feature maps of layer conv5 4 and below, there are no restrictions on the input image size. During the experiment, when we are extracting features from layer fc6-conv and above, the minimum size of an image is set to be 224 if it is less than 224.
APPENDIX B THE DETAIL OF OVERLAP IN EACH SCALE
In this paper, the overlaps between different regions occur in the 3 and 4 scale pyramid. A single region in each scale can be specified as the combination of a slice from the the width and height of the feature map. If a scale has N × N regions, then the number of slices in width and height of the feature map are both N . We use the same set of slices for both the width and height in this experiment.
In 3 scale (see Table 3 (b3)), overlap occurs only in scale 2, and the slice (in the proportion to the length of feature map width or height: {(0, 23 ), ( 1 3 , 1)}. In 4 scale v1 (Table 3 (c1)–(c3)), the slices for scale 2 and 3 are {(0, 34 ), ( 1 4 , 1)} and {(0, 2 4 ), ( 1 4 , 3 4 ), ( 2 4 , 1)}. In 4 scale v2 (Table 3 (c4)(c5)), the slices for scale 2 and 3 are {(0, 35 ), ( 2 5 , 1)} and {(0, 3 5 ), ( 1 5 , 4 5 ), ( 2 5 , 1)}. In 4 scale v3 (Table 3 (c6)–(c8)), the slices are {(0, 46 ), ( 2 6 , 1)} and {(0, 3 6 ), ( 1 6 , 4 6 ), ( 3 6 , 1)}, for scale 2 and 3, respectively.

5)
I agree with other reviewers on the lack of comparison with Gordo et al and Radenovic et al, though I understand that authors' arguments are that they do not want to train their networks (though then comparing with Arandjelovic et al. also doesn't make sense). It's still worth citing these papers and commenting on them. Also, with such an extensive set of experiments, it's a bit arguable if authors don't really do training - they don't do the canonical SGD, but they essentially perform grid search for parameters on the test (see question 3).
6)
I'm not sure what did we actually learn from this paper. To use the last conv? We knew that before as all recent papers do this (Arandjelovic et al, Tolias et al, Gordo et al, Radenovic et al, Babenko and Lempitsky, ..). That using original image sizes is important? We knew this as well, early works (Babenko et al 2014, etc) used smaller images while all recent works apply the networks convolutionally over original size images (e.g. Tolias et al have this experiment in table 1). That one should use PCA with whitening (and if possible learn whitening on the test set)? We knew this already as well. So the only two things that haven't been done in exactly the same way as people did it before is the multi-scale pooling (though obviously various other similar versions exist), and the exploration of max/sum pooling with l1 or l2 normalization (though the experiments in table 1 are basically ignored as sum-l1 works the best there, but authors then say that actually later they notice that for multiscale max-l2 works best). Actually the most interesting part for me, one that I can actually say I didn't know and don't think anyone knew, is figure 3.
7)
I think it's a bit of an overstatement to call this paper 'best practice for CNNs' when only a single CNN architecture, VGG-19, is considered. What is the best practice for other models, e.g. ResNet, Inception? Presumably the last conv is likely to be best though for ResNet it's not that clear, and I'm not sure if sum vs max pooling would change as those two networks were trained with sum pooling, and I'm not sure if any of the other conclusions hold either. This is more of a surgery of VGG-19 than best practices for CNNs in general.
8)
On a more philosophical level, and not only aimed at authors but also at others who are potentially reading this - this conference is about learning representations, while no learning is being performed. Taking CNNs as black boxes and tweaking the inputs and outputs in different ways with different normalizations is much more like using hand-engineered features like SIFT (replace black-box SIFT extractor with black-box CNN) than actually doing Deep Learning. I'm not saying this type of paper shouldn't exist as it's good to know what works best, but my preference in terms of what papers I would like to see in the future is:
a) There have been too many papers for using CNNs as black-boxes, I hoped we are finally over with this
b) For ICLR I think one should actually do some training, e.g. after we figure out the best image representation, now train the whole system end-to-end and see if you can improve the performance.
c) Design architectures which are specifically aimed at image retrieval - maybe something different than CNNs for classification pops up?
d) Figure out ways to train CNNs for retrieval, we know how to do it for classification by paying people to label millions of images, can we do something better for retrieval? (though this is to some extend addressed now by Arandjelovic et al, Gordo et al and Radenovic et al).
Other minor comments:
- I was also surprised by the "harder than category retrieval" statement, as reviewer 3. I wouldn't go as far as saying that the opposite is true either, the two just cannot be compared so easily.
- Inconsistencies of references (e.g. "Y. Lecun" vs "Ross Girshick", "CVPR" versus "Computer
Vision and Pattern Recognition", ..


SAMPLERNN: AN UNCONDITIONAL END-TO-END NEURAL AUDIO GENERATION MODEL
In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.
1 INTRODUCTION
Audio generation is a challenging task at the core of many problems of interest, such as text-tospeech synthesis, music synthesis and voice conversion. The particular difficulty of audio generation is that there is often a very large discrepancy between the dimensionality of the the raw audio signal and that of the effective semantic-level signal. Consider the task of speech synthesis, where we are typically interested in generating utterances corresponding to full sentences. Even at a relatively low sample rate of 16kHz, on average we will have 6,000 samples per word generated. 1
Traditionally, the high-dimensionality of raw audio signal is dealt with by first compressing it into spectral or hand-engineered features and defining the generative model over these features. However, when the generated signal is eventually decompressed into audio waveforms, the sample quality is often degraded and requires extensive domain-expert corrective measures. This results in complicated signal processing pipelines that are to adapt to new tasks or domains. Here we propose a step in the direction of replacing these handcrafted systems.
In this work, we investigate the use of recurrent neural networks (RNNs) to model the dependencies in audio data. We believe RNNs are well suited as they have been designed and are suited solutions for these tasks (see Graves (2013), Karpathy (2015), and Siegelmann (1999)). However, in practice it is a known problem of these models to not scale well at such a high temporal resolution as is found when generating acoustic signals one sample at a time, e.g., 16000 times per second. This is one of the reasons that Oord et al. (2016) profits from other neural modules such as one presented by Yu & Koltun (2015) to show extremely good performance.
In this paper, an end-to-end unconditional audio synthesis model for raw waveforms is presented while keeping all the computations tractable.2 Since our model has different modules operating at different clock-rates (which is in contrast to WaveNet), we have the flexibility in allocating the amount of computational resources in modeling different levels of abstraction. In particular, we can potentially allocate very limited resource to the module responsible for sample level alignments
1Statistics based on the average speaking rate of a set of TED talk speakers http://sixminutes. dlugan.com/speaking-rate/
2Code https://github.com/soroushmehr/sampleRNN_ICLR2017 and samples https:// soundcloud.com/samplernn/sets
operating at the clock-rate equivalent to sample-rate of the audio, while allocating more resources in modeling dependencies which vary very slowly in audio, for example identity of phoneme being spoken. This advantage makes our model arbitrarily flexible in handling sequential dependencies at multiple levels of abstraction.
Hence, our contribution is threefold:
1. We present a novel method that utilizes RNNs at different scales to model longer term dependencies in audio waveforms while training on short sequences which results in memory efficiency during training.
2. We extensively explore and compare variants of models achieving the above effect. 3. We study and empirically evaluate the impact of different components of our model on
three audio datasets. Human evaluation also has been conducted to test these generative models.
2 SAMPLERNN MODEL
In this paper we propose SampleRNN (shown in Fig. 1), a density model for audio waveforms. SampleRNN models the probability of a sequence of waveform samples X = {x1, x2, . . . , xT } (a random variable over input data sequences) as the product of the probabilities of each sample conditioned on all previous samples:
p(X) = T−1∏ i=0 p(xi+1|x1, . . . , xi) (1)
RNNs are commonly used to model sequential data which can be formulated as:
ht = H(ht−1, xi=t) (2) p(xi+1|x1, . . . , xi) = Softmax(MLP (ht)) (3)
with H being one of the known memory cells, Gated Recurrent Units (GRUs) (Chung et al., 2014), Long Short Term Memory Units (LSTMs) (Hochreiter & Schmidhuber, 1997), or their deep variations (Section 3). However, raw audio signals are challenging to model because they contain structure at very different scales: correlations exist between neighboring samples as well as between ones thousands of samples apart.
SampleRNN helps to address this challenge by using a hierarchy of modules, each operating at a different temporal resolution. The lowest module processes individual samples, and each higher module operates on an increasingly longer timescale and a lower temporal resolution. Each module conditions the module below it, with the lowest module outputting sample-level predictions. The entire hierarchy is trained jointly end-to-end by backpropagation.
2.1 FRAME-LEVEL MODULES
Rather than operating on individual samples, the higher-level modules in SampleRNN operate on non-overlapping frames of FS(k) (“Frame Size”) samples at the kth level up in the hierarchy at a time (frames denoted by f (k)). Each frame-level module is a deep RNN which summarizes the history of its inputs into a conditioning vector for the next module downward.
The variable number of frames we condition upon up to timestep t−1 is expressed by a fixed length hidden state or memory h(k)t where t is related to clock rate at that tier. The RNN makes a memory update at timestep t as a function of the previous memory h(k)t−1 and an input inp (k) t . This input for top tier k = K is simply the input frame. For intermediate tiers (1 < k < K) this input is a linear combination of conditioning vector from higher tier and current input frame. See Eqs. 4–5.
Because different modules operate at different temporal resolutions, we need to upsample each vector c at the output of a module into a series of r(k) vectors (where r(k) is the ratio between the temporal resolutions of the modules) before feeding it into the input of the next module downward (Eq. 6). We do this with a set of r(k) separate linear projections.
Here we are formalizing the frame-level module in tier k. Note that following equations are exclusive to tier k and timestep t for that specific tier. To increase the readability, unless necessary superscript (k) is not shown for t, inp(k), W (k)x , h(k),H(k), W (k)j , and r(k).
inpt =
{ Wxf (k) t + c (k+1) t ; 1 < k < K
f (k=K) t ; k = K
(4)
ht = H(ht−1, inpt) (5)
c (k) (t−1)∗r+j =Wjht; 1 ≤ j ≤ r (6)
Our approach of upsampling with r(k) linear projections is exactly equivalent to upsampling by adding zeros and then applying a linear convolution. This is sometimes called “perforated” upsampling in the context of convolutional neural networks (CNNs). It was first demonstrated to work well in Dosovitskiy et al. (2016) and is a fairly common upsampling technique.
2.2 SAMPLE-LEVEL MODULE
The lowest module (tier k = 1; Eqs. 7–9) in the SampleRNN hierarchy outputs a distribution over a sample xi+1, conditioned on the FS(1) preceding samples as well as a vector c (k=2) i from the next higher module which encodes information about the sequence prior to that frame. As FS(1) is usually a small value and correlations in nearby samples are easy to model by a simple memoryless module, we implement it with a multilayer perceptron (MLP) rather than RNN which slightly speeds up the training. Assuming ei represents xi after passing through embedding layer (section 2.2.1), conditional distribution in Eq. 1 can be achieved by following and for further clarity two consecutive sample-level frames are shown. In addition, Wx in Eq. 8 is simply used to linearly combine a frame and conditioning vector from above.
f (1) i−1 = flatten([ei−FS(1) , . . . , ei−1]) (7)
f (1) i = flatten([ei−FS(1)+1, . . . , ei])
inp (1) i =W (1) x f (1) i + c (2) i (8)
p(xi+1|x1, . . . , xi) = Softmax(MLP (inp(1)i )) (9)
We use a Softmax because we found that better results were obtained by discretizing the audio signals (also see van den Oord et al. (2016)) and outputting a Multinoulli distribution rather than using a Gaussian or Gaussian mixture to represent the conditional density of the original real-valued signal. When processing an audio sequence, the MLP is convolved over the sequence, processing
each window of FS(1) samples and predicting the next sample. At generation time, the MLP is run repeatedly to generate one sample at a time. Table 1 shows a considerable gap between the baseline model RNN and this model, suggesting that the proposed hierarchically structured architecture of SampleRNN makes a big difference.
2.2.1 OUTPUT QUANTIZATION
The sample-level module models its output as a q-way discrete distribution over possible quantized values of xi (that is, the output layer of the MLP is a q-way Softmax).
To demonstrate the importance of a discrete output distribution, we apply the same architecture on real-valued data by replacing the q-way Softmax with a Gaussian Mixture Models (GMM) output distribution. Table 2 shows that our model outperforms an RNN baseline even when both models use real-valued outputs. However, samples from the real-valued model are almost indistinguishable from random noise.
In this work we use linear quantization with q = 256, corresponding to a per-sample bit depth of 8. Unintuitively, we realized that even linearly decreasing the bit depth (resolution of each audio sample) from 16 to 8 can ease the optimization procedure while generated samples still have reasonable quality and are artifact-free.
In addition, early on we noticed that the model can achieve better performance and generation quality when we embed the quantized input values before passing them through the sample-level MLP (see Table 4). The embedding steps maps each of the q discrete values to a real-valued vector embedding. However, real-valued raw samples are still used as input to the higher modules.
2.2.2 CONDITIONALLY INDEPENDENT SAMPLE OUTPUTS
To demonstrate the importance of a sample-level autoregressive module, we try replacing it with “Multi-Softmax” (see Table 4), where the prediction of each sample xi depends only on the conditioning vector c from Eq. 9. In this configuration, the model outputs an entire frame of FS(1) samples at a time, modeling all samples in a frame as conditionally independent of each other. We find that this Multi-Softmax model (which lacks a sample-level autoregressive module) scores significantly worse in terms of log-likelihood and fails to generate convincing samples. This suggests that modeling the joint distribution of the acoustic samples inside each frame is very important in order to obtain good acoustic generation. We found this to be true even when the frame size is reduced, with best results always with a frame size of 1, i.e., generating only one acoustic sample at a time.
2.3 TRUNCATED BPTT
Training recurrent neural networks on long sequences can be very computationally expensive. Oord et al. (2016) avoid this problem by using a stack of dilated convolutions instead of any recurrent connections. However, when they can be trained efficiently, recurrent networks have been shown to be very powerful and expressive sequence models. We enable efficient training of our recurrent model using truncated backpropagation through time, splitting each sequence into short subsequences and propagating gradients only to the beginning of each subsequence. We experiment with different subsequence lengths and demonstrate that we are able to train our networks, which model very long-term dependencies, despite backpropagating through relatively short subsequences.
Table 3 shows that by increasing the subsequence length, performance substantially increases alongside with train-time memory usage and convergence time. Yet it is noteworthy that our best models have been trained on subsequences of length 512, which corresponds to 32 milliseconds, a small fraction of the length of a single a phoneme of human speech while generated samples exhibit longer word-like structures.
Despite the aforementioned fact, this generative model can mimic the existing long-term structure of the data which results in more natural and coherent samples that is preferred by human listeners. (More on this in Sections 3.2–3.3.) This is due to the fast updates from TBPTT and specialized frame-level modules (Section 2.1) with top tiers designed to model a lower resolution of signal while leaving the process of filling the details to lower tiers.
3 EXPERIMENTS AND RESULTS
In this section we are introducing three datasets which have been chosen to evaluate the proposed architecture for modeling raw acoustic sequences. The description of each dataset and their preprocessing is as follows:
Blizzard which is a dataset presented by Prahallad et al. (2013) for speech synthesis task, contains 315 hours of a single female voice actor in English; however, for our experiments we are using only 20.5 hours. The training/validation/test split is 86%-7%-7%. Onomatopoeia3, a relatively small dataset with 6,738 sequences adding up to 3.5 hours, is human vocal sounds like grunting, screaming, panting, heavy breathing, and coughing. Diversity of sound type and the fact that these sounds were recorded from 51 actors and many categories makes it a challenging task. To add to that, this data is extremely unbalanced. The training/validation/test split is 92%-4%-4%. Music dataset is the collection of all 32 Beethoven’s piano sonatas publicly available on https://archive.org/ amounting to 10 hours of non-vocal audio. The training/validation/test split is 88%-6%-6%.
See Fig. 2 for a visual demonstration of examples from datasets and generated samples. For all the datasets we are using a 16 kHz sample rate and 16 bit depth. For the Blizzard and Music datasets, preprocessing simply amounts to chunking the long audio files into 8 seconds long sequences on which we will perform truncated backpropagation through time. Each sequence in the Onomatopoeia dataset is few seconds long, ranging from 1 to 11 seconds. To train the models on this dataset, zero-padding has been applied to make all the sequences in a mini-batch have the same length and corresponding cost values (for the predictions over the added 0s) would be ignored when computing the gradients.
We particularly explored two gated variants of RNNs—GRUs and LSTMs. For the case of LSTMs, the forget gate bias is initialized with a large positive value of 3, as recommended by Zaremba (2015) and Gers (2001), which has been shown to be beneficial for learning long-term dependencies.
As for models that take real-valued input, e.g. the RNN-GMM and SampleRNN-GMM (with 4 components), normalization is applied per audio sample with the global mean and standard deviation obtained from the train split. For most of our experiments where the model demands discrete input, binning was applied per audio sample.
All the models have been trained with teacher forcing and stochastic gradient decent (mini-batch size 128) to minimize the Negative Log-Likelihood (NLL) in bits per dimension (per audio sample). Gradients were hard-clipped to remain in [-1, 1] range. Update rules from the Adam optimizer (Kingma & Ba, 2014) (β1 = 0.9, β2 = 0.999, and = 1e−8) with an initial learning rate of 0.001 was used to adjust the parameters. For training each model, random search over hyper-parameter values (Bergstra & Bengio, 2012) was conducted. The initial RNN state of all the RNN-based models was always learnable. Weight Normalization (Salimans & Kingma, 2016) has been used for all the linear layers in the model (except for the embedding layer) to accelerate the training procedure. Size of the embedding layer was 256 and initialized by standard normal distribution. Orthogonal weight matrices used for hidden-to-hidden connections and other weight matrices initialized similar to He et al. (2015). In final model, we found GRU to work best (slightly better than LSTM). 1024 was the the number of hidden units for all GRUs (1 layer per tier for 3-tier and 3 layer for 2-tier model) and MLPs (3 fully connected layers with ReLU activation with output dimension being 1024 for first two layers and 256 for the final layer before softmax). Also FS(1) = FS(2) = 2 and FS(3) = 8 were found to result in lowest NLL.
3.1 WAVENET RE-IMPLEMENTATION
We implemented the WaveNet architecture as described in Oord et al. (2016). Ideally, we would have liked to replicate their model exactly but owing to missing details of architecture and hyperparameters, as well as limited compute power at our disposal, we made our own design choices so that the model would fit on a single GPU while having a receptive field of around 250 milliseconds,
3Courtesy of Ubisoft
Table 3: Effect of subsequence length on NLL (bits per audio sample) computed on the Blizzard validation set. Subsequence Length 32 64 128 256 512
NLL Validation 1.575 1.468 1.412 1.391 1.364
while having a reasonable number of updates per unit time. Although our model is very similar to WaveNet, the design choices, e.g. number of convolution filters in each dilated convolution layer, length of target sequence to train on simultaneously (one can train with a single target with all samples in the receptive field as input or with target sequence length of size T with input of size receptive field + T - 1), batch-size, etc. might make our implementation different from what the authors have done in the original WaveNet model. Hence, we note here that although we did our best at exactly reproducing their results, there would very likely be different choice of hyper-parameters between our implementation and the one of the authors.
For our WaveNet implementation, we have used 4 dilated convolution blocks each having 10 dilated convolution layers with dilation 1, 2, 4, 8 up to 512. Hence, our network has a receptive field of 4092 acoustic samples i.e. the parameters of multinomial distribution of sample at time step t, p(xi) = fθ(xi−1, xi−2, . . . xi−4092) where θ is model parameters. We train on target sequence length of 1600 and use batch size of 8. Each dilated convolution filter has size 2 and the number of output channels is 64 for each dilated convolutional layer (128 filters in total due to gated nonlinearity). We trained this model using Adam optimizer with a fixed global learning rate of 0.001 for Blizzard dataset and 0.0001 for Onomatopoeia and Music datasets. We trained these models for about one week on a GeForce GTX TITAN X. We dropped the learning rate in the Blizzard experiment to 0.0001 after around 3 days of training.
3.2 HUMAN EVALUATION
Apart from reporting NLL, we conducted AB preference tests for random samples from four models trained on the Blizzard dataset. For unconditional generation of speech which at best sounds like mumbling, this type of test is the one which is more suited. Competing models were the RNN, SampleRNN (2-tier), SampleRNN (3-tier), and our implementation of WaveNet. The rest of the models were excluded as the quality of samples were definitely lower and also to keep the number of pair comparison tests manageable. We will release the samples that have been used in this test too.
All the samples were set to have the same volume. Every user is then shown a set of twenty pairs of samples with one random pair at a time. Each pair had samples from two different models. The human evaluator is asked to listen to the samples and had the option of choosing between the two model or choosing not to prefer any of them. Hence, we have a quantification of preference between every pair of models. We used the online tool made publicly available by Jillings et al. (2015).
Results in Fig. 3 clearly points out that SampleRNN (3-tier) is a winner by a huge margin in terms of preference by human raters, then SampleRNN (2-tier) and afterward two other models, which matches with the performance comparison in Table 1.
The same evaluation was conducted for Music dataset except for an additional filtering process of samples. Specific to only this dataset, we observed that a batch of generated samples from competing models (this time restricted to RNN, SampleRNN (2-tier), and SampleRNN (3-tier)) were either music-like or random noise. For all these models we only considered random samples that were not random noise. Fig. 4 is dedicated to result of human evaluation on Music dataset.
3.3 QUANTIFYING INFORMATION RETENTION
For the last experiment we are interested in measuring the memory span of the model. We trained our model, SampleRNN (3-tier), with best hyper-parameters on a dataset of 2 speakers reading audio books, one male and one female, respectively, with mean fundamental frequency of 125.3 and 201.8Hz. Each speaker has roughly 10 hours of audio in the dataset that has been preprocessed similar to Blizzard. We observed that it learned to stay consistent generating samples from the same speaker without having any knowledge about the speaker ID or any other conditioning information. This effect is more apparent here in comparison to the unbalanced Onomatopoeia that sometimes mixes two different categories of sounds.
Another experiment was conducted to test the effect of memory and study the effective memory horizon. We inject 1 second of silence in the middle of sampling procedure in order to see if it will remember to generate from the same speaker or not. Initially when sampling we let the model generate 2 seconds of audio as it normally do. From 2 to 3 seconds instead of feeding back the generated sample at that timestep a silent token (zero amplitude) would be fed. From 3 to 5 seconds again we sample normally; feeding back the generated token.
We did classification based on mean fundamental frequency of speakers for the first and last 2 seconds. In 83% of samples SampleRNN generated from the same person in two separate segments.
This is in contrast to a model with fixed past window like WaveNet where injecting 16000 silent tokens (3.3 times the receptive field size) is equivalent to generating from scratch which has 50% chance (assuming each 2-second segment is coherent and not a mixed sound of two speakers).
4 RELATED WORK
Our work is related to earlier work on auto-regressive multi-layer neural networks, starting with Bengio & Bengio (1999), then NADE (Larochelle & Murray, 2011) and more recently PixelRNN (van den Oord et al., 2016). Similar to how they tractably model joint distribution over units of the data (e.g. words in sentences, pixels in images, etc.) through an auto-regressive decomposition, we transform the joint distribution of acoustic samples using Eq. 1.
The idea of having part of the model running at different clock rates is related to multi-scale RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutnik et al., 2014; Sordoni et al., 2015; Serban et al., 2016).
Chung et al. (2015) also attempt to model raw audio waveforms which is in contrast to traditional approaches which use spectral features as in Tokuda et al. (2013), Bertrand et al. (2008), and Lee et al. (2009).
Our work is closely related to WaveNet (Oord et al., 2016), which is why we have made the above comparisons, and makes it interesting to compare the effect of adding higher-level RNN stages working at a low resolution. Similar to this work, our models generate one acoustic sample at a time conditioned on all previously generated samples. We also share the preprocessing step of quantizing the acoustics into bins. Unlike this model, we have different modules in our models running at different clock-rates. In contrast to WaveNets, we mitigate the problem of long-term dependency with hierarchical structure and using stateful RNNs, i.e. we will always propagate hidden states to the next training sequence although the gradient of the loss will not take into account the samples in previous training sequence.
5 DISCUSSION AND CONCLUSION
We propose a novel model that can address unconditional audio generation in the raw acoustic domain, which typically has been done until recently with hand-crafted features. We are able to show that a hierarchy of time scales and frequent updates will help to overcome the problem of modeling extremely high-resolution temporal data. That allows us, for this particular application, to learn the data manifold directly from audio samples. We show that this model can generalize well and generate samples on three datasets that are different in nature. We also show that the samples generated by this model are preferred by human raters.
Success in this application, with a general-purpose solution as proposed here, opens up room for more improvement when specific domain knowledge is applied. This method, however, proposed with audio generation application in mind, can easily be adapted to other tasks that require learning the representation of sequential data with high temporal resolution and long-range complex structure.
ACKNOWLEDGMENTS
The authors would like to thank João Felipe Santos and Kyle Kastner for insightful comments and discussion. We would like to thank the Theano Development Team (2016)4 and MILA staff. We acknowledge the support of the following agencies for research funding and computing support: NSERC, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. Jose Sotelo also thanks the Consejo Nacional de Ciencia y Tecnologı́a (CONACyT) as well as the Secretarı́a de Educación Pública (SEP) for their support. This work was a collaboration with Ubisoft.
4http://deeplearning.net/software/theano/
APPENDIX A
A MODEL VARIANT: SAMPLERNN-WAVENET HYBRID
SampleRNN-WaveNet model has two modules operating at two different clock-rate. The slower clock-rate module (frame-level module) sees one frame (each of which has size FS) at a time while the faster clock-rate component(sample-level component) sees one acoustic sample at a time i.e. the ratio of clock-rates for these two modules would be the size of a single frame. Number of sequential steps for frame-level component would be FS times lower. We repeat the output of each step of frame-level component FS times so that number of time-steps for output of both the components match. The output of both these modules are concatenated for every time-step which is further operated by non-linearities for every time-step independently before generating the final output.
In our experiments, we kept size of a single frame (FS) to be 128. We tried two variants of this model: 1. fully convolutional WaveNet and 2. RNN-WaveNet. In fully convolutional WaveNet, both modules described above are implemented using dilated convolutions as described in original WaveNet model. In RNN-WaveNet, we use high capacity RNN in the frame-level module to model the dependency between frames. The sample-level WaveNet in RNN-WaveNet has receptive field of size 509 samples from the past.
Although these models are designed with the intention of combining the two models to harness their best features, preliminary experiments show that this variant is not meeting our expectations at the moment which directs us to a possible future work.

The paper introduces SampleRNN, a hierarchical recurrent neural network model of raw audio. The model is trained end-to-end and evaluated using log-likelihood and by human judgement of unconditional samples, on three different datasets covering speech and music. This evaluation shows the proposed model to compare favourably to the baselines.
It is shown that the subsequence length used for truncated BPTT affects performance significantly, but interestingly, a subsequence length of 512 samples (~32 ms) is sufficient to get good results, even though the features of the data that are modelled span much longer timescales. This is an interesting and somewhat unintuitive result that I think warrants a bit more discussion.
The authors have attempted to reimplement WaveNet, an alternative model of raw audio that is fully convolutional. They were unable to reproduce the exact model architecture from the original paper, but have attempted to build an instance of the model with a receptive field of about 250ms that could be trained in a reasonable time using their computational resources, which is commendable.
The architecture of the Wavenet model is described in detail, but it found it challenging to find the same details for the proposed SampleRNN architecture (e.g. which value of "r" is used for the different tiers, how many units per layer, ...). I think a comparison in terms of computational cost, training time and number of parameters would also be very informative.
Surprisingly, Table 1 shows a vanilla RNN (LSTM) substantially outperforming this model in terms of likelihood, which is quite suspicious as LSTMs tend to have effective receptive fields of a few hundred timesteps at best. One would expect the much larger receptive field of the Wavenet model to be reflected in the likelihood scores to some extent. Similarly, Figure 3 shows the vanilla RNN outperforming the Wavenet reimplementation in human evaluation on the Blizzard dataset. This raises questions about the implementation of the latter. Some discussion about this result and whether the authors expected it or not would be very welcome.
Table 1 and Figure 4 also show the 2-tier SampleRNN outperforming the 3-tier model in terms of likelihood and human rating respectively, which is very counterintuitive as one would expect longer-range temporal correlations to be even more relevant for music than for speech. This is not discussed at all, I think it would be useful to comment on why this could be happening.
Overall, this an interesting attempt to tackle modelling very long sequences with long-range temporal correlations and the results are quite convincing, even if the same can't always be said of the comparison with the baselines. It would be interesting to see how the model performs for conditional generation, seeing as it can be more easily be objectively compared to models like Wavenet in that domain.
Other remarks:
- upsampling the output of the models is done with r separate linear projections. This choice of upsampling method is not motivated. Why not just use linear interpolation or nearest neighbour upsampling? What is the advantage of learning this operation? Don't the r linear projections end up learning largely the same thing, give or take some noise?
- The third paragraph of Section 2.1.1 indicates that 8-bit linear PCM was used. This is in contrast to Wavenet, for which an 8-bit mu-law encoding was used, and this supposedly improves the audio fidelity of the samples. Did you try this as well?
- Section 2.1 mentions the discretisation of the input and the use of a softmax to model this discretised input, without any reference to prior work that made the same observation. A reference is given in 2.1.1, but it should probably be moved up a bit to avoid giving the impression that this is a novel observation.