Most of the pedestrian detection methods are based on hand-crafted features which produce low accuracy on complex scenes. With the development of deep learning method, pedestrian detection has achieved great success. In this paper, we take advantage of a convolutional neural network which is based on Fast R-CNN framework to extract robust pedestrian features for efficient and effective pedestrian detection in complicated environments. We use the EdgeBoxes algorithm to generate effective region proposals from an image, as the quality of extracted region proposals can greatly affect the detection performance. In order to reduce the training time and to improve the generalization performance, we add a batch normalization layer between the convolutional layer and the activation function layer. Experiments show that the proposed method achieves satisfactory performance on the INRIA and ETH datasets.
1 INTRODUCTION
In recent years, pedestrian detection has become an important branch of object detection and has attracted wide attention in computer vision Dalal & Triggs (2005); Dollár et al. (2014; 2009); Viola et al. (2005); Wang et al. (2009). In real life, it is a key problem in automotive safety, video surveillance, smart vehicles and intelligent robotics. Because of the diversity of pedestrian body pose, object occlusions, clothing, lighting change and complicated backgrounds in the video sequence or image, the pedestrian detection is still a challenging task in computer vision.
In pedestrian detection, feature extraction is an important factor to influent the performance. Many features have been proposed by researchers for pedestrian detection, such as Haar-like features Viola & Jones (2004), Integral Channel Features (ICF) Dollár et al. (2009), Histogram of Oriented Gradients (HOG) Dalal & Triggs (2005), Local Binary Pattern (LBP) Mu et al. (2008), Dense SIFT Vedaldi et al. (2009) etc. And the proposed Deformable Part Based Model (DPM) Felzenszwalb et al. (2010) which is based on HOG, has maken a breakthrough in pedestrian detection. However, these features are hand-craft and are considered to be low-level. They use the low-level information while the high-level information is usually very important for pedestrian detection Sermanet et al. (2013).
Recently, with the development of deep learning techniques, deep neural networks have been successfully applied in object recognition tasks Girshick et al. (2014); He et al. (2015); Girshick (2015); Zhao et al. (2014). Deep neural networks can achieve more excellent results due to their capability to learn discriminative features from raw pixels. Many researchers have applied deep learning techniques to pedestrian detection. Sermanet et al. Sermanet et al. (2013) proposed a two layers convolutional model which adopts convolutional sparse coding to pre-train convolutional neural network for pedestrian detection. Chen et al. Chen et al. (2014) proposed a pre-trained Deep Convolutional Neural Network(DCNN) to learn features from ACF Dollár et al. (2014) detector. These
features are then fed to a SVM classifier. Ouyang et al. Ouyang & Wang (2013) proposed a joint deep model that jointly learns four key components in pedestrian detection: feature extraction, deformation handling, occlusion handling and classifier. Li et al. Li et al. (2015) proposed a SAF R-CNN which takes scales of pedestrians into account in pedestrian detection. Fast R-CNN Girshick (2015) is one of the best detection approaches based on deep leaning. It determines regions of interest(RoI) in an image using superpixel method, and then extracts features using convolutional neural network model. The extracted features are then passed to softmax layer and bounding box regressor layer which are trained for each object, and finally the pedestrian and its position are detected.
It is well known that training a deep neural network is complicated due to the fact that internal covariate shift Ioffe & Szegedy (2015) can degrade the efficiency of training. Internal covariate shift is a change of distribution of the inputs. Which happens during training the feed-forward neural networks, by changing the parameters of a layer. Batch normalization (BN) Ioffe & Szegedy (2015) is a technique to control the distributions of feed-forward neural network activations, thereby reducing internal covariate shift. So deep neural networks trained with batch normalization can converge faster and generalize better. In this paper, we use Fast R-CNN architecture based on batch normalization(BN) to extract robust pedestrian features.
In pedestrian detection task, another important stage is locating the potential windows which may contain pedestrians. As an exhaustive and traditional method, sliding window method has two main shortcomings. First, it needs searching every possible position in an image. Second, it may produce many redundancy windows which affect the quality of detection. To improve the detection efficiency, the approaches of region proposals have been proposed to produce high quality regions, in which the most popular method is Selective Search Uijlings et al. (2013). It can get high quality of regions by using image segmentation. However, its speed is very slow and the detection performance is sensitive to the image quality. And the Selective Search method of extracting candidate windows is infeasible Chen et al. (2014) but cannot provide precise localization of pedestrians. To produce more precise localized candidate windows, we employ EdgeBoxes Zitnick & Dollár (2014) method, which is a fast and effective one Hosang et al. (2014). It is believed that the edge information can precisely describe the object, so the EdgeBoxes method can extract higher quality of the candidate windows than the Selective Search method. Moreover, the EdgeBoxes method actually performs better in terms of the average intersection over union(IoU) across all images in the set Hosang et al. (2015).
In this paper, our main contributions are summarized as follows:
(1) In order to make the pedestrian detection more efficient, we use the EdgeBoxes method which can obtain low-redundancy and high quality of candidate windows.
(2) We use the Fast R-CNN architecture to extract robust features for pedestrian detection. In order to reduce the training time and prevent the gradients that explode or vanish when training the Fast R-CNN, we add the batch normalization(BN) layer into the Fast R-CNN architecture.
The rest of this paper can be organized as follows. In Section 2, we introduce the related work of pedestrian detection and briefly introduce the batch normalization layer. In Section 3, we introduce our pedestrian detection approach. In Section 4, we present our experiment results on two benchmark datasets, and some analyses as well. In Section 5, we conclude our work and discuss the possible advances to our model in the future.
2 RELATED WORK
In this section, we review the related work of two important stages of pedestrian detection, viz. extracting region proposals and extracting features, respectively. And, we also briefly introduce the batch normalization layer.
Region Proposals: In an image, the position of a pedestrian can be anywhere and its size can be arbitrary, so it is necessary to search the whole image to localize the pedestrians. Traditional methods employ sliding window method to find the possible locations on the whole image. The sliding window method can get almost all potential locations through exhaustive search, but it is computational and may produce many redundancy windows. Some region proposals methods were proposed to overcome this problem Hosang et al. (2015), in which the most popular method is
Selective Search Uijlings et al. (2013). It is a combination of exhaustive search and segmentation method. Wang et al. Wang et al. (2015) proposed a region proposal fusion algorithm to fusion the Selective Search and BING Cheng et al. (2014) method. Nam et al. Nam et al. (2014) proposed LDCF method to extract regions for pedestrian detection. The EdgeBoxes method has been shown as a state-of-the-art region proposal system Zitnick & Dollár (2014); Hosang et al. (2014).
Extracting features: Many models based on hand-crafted features have been utilized for pedestrian detection Dalal & Triggs (2005); Dollár et al. (2009); Wang et al. (2009); Viola & Jones (2004); Mu et al. (2008). Wang et al. Wang et al. (2009) utilized the combination of HOG and LBP to handle the partial occlusion of pedestrian. Felzenswalb et al. Mu et al. (2008) further improved the detection performance by combining the HOG with a deformable part model. Dollár et al. proposed the Integral Channel Features(ICF) Dollár et al. (2009) and Aggregated Channel Features(ACF) Dollár et al. (2014) which efficiently extract histograms and haar features. Chen et al. Chen et al. (2013) used a multi-order context representation to take advantage of co-occurrence contexts of different objects. Deep learning models can be trained end-to-end to extract robust features from the given images. Sermanet et al. Sermanet et al. (2013) proposed a convolutional neural network model which has two convolutional layers pre-trained by sparse coding. Ouyang et al. Ouyang & Wang (2013) proposed a joint deep model which contains a deformation layer to model mixture poses information. Tian et al. Tian et al. (2015) jointly utilized semantic tasks to optimize pedestrian detection. More recently, some excellent works have been successfully applied in object detection Girshick et al. (2014); He et al. (2015); Girshick (2015), and the performance of pedestrian detection is largely improved Chen et al. (2014); Li et al. (2015); Wang et al. (2015). Fast R-CNN Girshick (2015) is one of the best detection approaches among them. It consists of two steps for object detection, in which the first step is to use non-deep learning methods to extract thousands of region proposals, and the second step is to use convolutional neural networks classifiers to classify categories at those locations.
Batch Normalization: Training a deep neural network is complicated due to the fact that changing the parameters of a layer will affect the distribution of the inputs to the succedent layers. As a result, the succedent layers are continually adapted to fit the shifting input distribution and unable to learn effectively. This phenomenon is called Internal Covariate Shift (ICS) Ioffe & Szegedy (2015). The ICS slows down the course of training and needs careful parameter initialization. Batch normalization(BN) Ioffe & Szegedy (2015) is a technique to accelerate training and to improve generalization capability. Its basic idea is to standardize the internal representations inside the network to make the network converging faster and generalizing better, inspired by the way to whiten the network input to improve performance. Batch normalization layer can be applied to convolutional neural networks directly between the convolutional layer and the activation function layer, for example ReLU etc. It uses the local mean and variance computed over the minibatch x, and then corrects with a learned variance and bias term, namely γ and β:
BN(x; γ, β) = β + γ x− E[x]
(V ar[x] + )1/2 (1)
where is a small positive constant to improve numerical stability.
3 OUR APPROACH
In this section, we will introduce the details of our approach for pedestrian detection. Firstly, we will introduce the proposal generation algorithm. Then, we will describe the pedestrian detection architecture. Finally, we will briefly introduce the whole pedestrian detection process.
3.1 PEDESTRIAN PROPOSALS
We utilize the EdgeBoxes as our proposal generation algorithm. The basic idea of EdgeBoxes is that it generates and scores the proposal based on the edge map of the given image.
Specifically, firstly, a structured edge detector is utilized to generate an edge map where each pixel contains a magnitude and orientation information of the edge. Secondly, the edges are grouped together by a greedy algorithm. Then, the affinity between two edge groups is computed. This is the input to the next step for the score computation. Thirdly, for a given bounding box, the score of the
bounding box is computed based on the edge groups entirely inside the box. Finally, as the EdgeBoxes detector may generate many overlapping candidate pedestrian windows, the non-maximal suppression is used to filter out these overlapping bounding boxes by the following criterion:
OverlapArea = (IntersectionArea)
(UnionArea) (2)
If two bounding boxes overlap by more than 50%, then the bounding box with the highest score is selected.
Compared with the Selective Search which is the most popular proposal method, EdgeBoxes is much faster. For EdgeBoxes, the average runtime is about 0.3 seconds per image while Selective Search is about 10 seconds per image. In the case of accuracy on the PASCAL VOC 2007 dataset, the mAP of Selective Search is 31.7% which is slightly smaller than the mAP of EdgeBoxes 31.8% Hosang et al. (2014), and on the INRIA dataset, the recall rates of Selective Search and EdgeBoxes are 23% and 93% respectively by our experiment. In brief, the EdgeBoxes can not only reduce the runtime complexity but also increase the accuracy. Therefore, we choose the EdgeBoxes as our proposal generation algorithm.
3.2 THE ARCHITECTURE FOR PEDESTRIAN DETECTION
The architecture of our proposed Fast R-CNN based on batch normalization method is illustrated in Figure 1. It has 7 layers. As batch normalization layer can normalize each scalar feature independently layer by layer, it avoids the gradient vanish and gradient explode problem. Batch normalization can lead to a faster convergence during the training and can improve the generalization performance as a regularization technique. In our proposed architecture, the batch normalization layer is placed between each convolutional layer and the succedent ReLU layer, as illustrated in Figure 2.
In the first convolutional layer, there are 96 kernels of size 11×11 with a stride of 4 pixels and in the following max-pooling layer, the kernel size is 3×3. The second convolutional layer takes as input the output of the first convolutional layer, and there are 256 kernels of size 5×5 with a stride of 2 pixels. In the following max-pooling layer, the kernel size is the same as the first layer. For the next two (the third and the fourth) convolutional layers, the corresponding pooling layers are omitted and both contain 384 kernels. In the fifth convolutional layer, there are 256 kernels and the following
pooling layer is a region of interest (RoI) pooling layer. The RoI pooling layer is utilized to pool the feature maps of each input object proposal into a fixed-length feature vector which is then fed into the fully connected layers. The next two layers are fully connected layers, both of which contain 4096 nodes. At the end of the architecture there are two output layers which produce two output vectors for per object proposal. Specifically, one is a softmax layer, which outputs classification scores over K object classes plus a “background” class. The other is a bounding-box regressor layer, which outputs four real-valued numbers for each of the K object classes. These 4×K values encode refined bounding boxes for each class.
3.3 PEDESTRIAN DETECTION PROCESS
The entire pedestrian detection process can be summarized as follows:
(1) Run the EdgeBoxes algorithm to generate region proposals.
(2) Input the whole image and a number of region proposals to our proposed model.
(3) Extract the features of region proposals with convolutional neural networks.
(4) Locate and verify pedestrian in each region proposal with extracted features by softmax classifier and bounding-box regressor.
The overall framework is also presented in Figure 3.
4 EXPERIMENTS AND ANALYSES
In this section, we evaluate our proposed pedestrian detection method on two popular benchmark datasets, INRIA and ETH. We follow the evaluation protocols proposed in Dollár et al. (2012). The log average miss rate is used to summarize the detector performance. It is computed by averaging the miss rate at 9 FPPI (false positives per image) rates (0.05, 0.10, 0.20, 0.30, 0.40, 0.50, 0.64, 0.80, 1) which are evenly spaced in the log-space ranging from 10−2 to 100. In one test image, if a detected bounding box BBdt can significantly cover most area of the body window BBgt of the ground-truth pedestrian, the detection window is considered to properly detect the pedestrian.
α0 = area(BBdt ∩BBgt) area(BBdt ∪BBgt) > θ (3)
where α0 is the area of overlap between BBdt and BBgt, and θ is the threshold which is set to be 0.5. And the experiments are run on an Ubuntu 14.04 64 bit system with a single 4GB memory NVIDIA GeForce GTX 980 GPU.
4.1 INRIA DATASET
The INRIA dataset is a very popular dataset for pedestrian detection. The annotations of pedestrians are of high quality in diverse conditions and poses. The details of INRIA dataset are listed in Table 1.
Our proposed model is implemented based on the publicly Caffe platform Jia et al. (2014) and the CaffeNet model is taken as our initial model which has been pre-trained on the ImageNet dataset. In
order to use the Fast R-CNN method, we replace the last max-pooling layer of the CaffeNet with the RoI pooling layer to pool the feature maps of each region proposal into a fixed resolution, i.e. 6×6. We also replace the final fully connected layer and softmax layer with two sibling fully connected layers, i.e. softmax layer and bounding-box regressor layer.
In order to expand positive example set, we map each region proposal to the ground-truth which has a high intersection over union (IoU). If the IoU is lager than 0.5, then we will label the selected region proposal as a positive sample for pedestrian class. The rest region proposals, whose IoUs are smaller than 0.1, are regarded as negative instances. The proposed model is trained with the Stochastic Gradient Descent (SGD) method with the momentum of 0.9 and the weight decay factor of 0.0005. Since batch normalization layer is added, a large learning rate is used in our experiment. We set the initial global learning rate as 0.01 and the dropout ratio as 0.1 in the fc6 and fc7 layers. The whole process of training is 40000 iterations, the training time of our model (FRCNN+BN) is 0.094 seconds per iteration and that of Fast R-CNN is 0.153 seconds per iteration. The average time to process an image using our model is about 0.5 seconds, while the average time of the Fast R-CNN is about 0.6 seconds.
The overall experimental results are reported with the DET curves (miss rate versus FPPI) in Figure 4. We compare our proposed model with existing methods, including two classical methods VJ Viola & Jones (2004), HOG Dalal & Triggs (2005) and other state-of-the-art methods such as ConvNet Sermanet et al. (2013), ACF Dollár et al. (2014), VeryFast Rodrigo et al. (2012), FRCNN Girshick (2015), Roerei Rodrigo et al. (2013), Sketch Tokens Joseph et al. (2013) and SpatialPooling Paisitkriangkrai et al. (2014). We also give the digitized results of the miss rate at 0.1 FPPI.
From Figure 4, we can see that our method outperforms most of the state-of-the-art methods. The miss rate of our method is 12%, which has an improvement of 8% over ConvNet and an improvement of 2% over FRCNN, but a slight decrease over the SpatialPooling method.
4.2 ETH DATASET
The ETH dataset derives from the binocular vision of pedestrian dataset. It is obtained by a pair of on-board camera and it gives the annotations of pedestrians. The set01, set02 and set03 are currently the most used three subsets. The details of these three subsets can be seen in Table 2.
In order to evaluate the generalization performance of our proposed method, we train our proposed method on the INRIA dataset, and then apply the trained model to the ETH dataset. The performance evaluation is the same as that on the INRIA dataset. The average time to process an image using our model (FRCNN+BN) is about 0.4 seconds, while the average time of the Fast R-CNN is about 0.5 seconds. We compare our model with other methods, such as ConvNet Sermanet et al. (2013), JointDeep Ouyang & Wang (2013), FRCNN Girshick (2015), SDN Luo et al. (2014), TA-CNN Tian et al. (2015), VJ Viola & Jones (2004), HOG Dalal & Triggs (2005), ACF Dollár et al. (2014), VeryFast Rodrigo et al. (2012). The comparison results are shown in Figure 5, from which we can see that our method outperforms VJ, HOG, VeryFast, ACF, ConvNet, FRCNN and JointDeep but underperforms SDN, SpatialPooling and TA-CNN methods.
From Figure 5, we can also see that the performance on the ETH dataset is worse than that on the INRIA dataset. One reason may lie in that the image quality of ETH dataset is much worse than that of the INRIA dataset, and the image resolution of the ETH dataset is lower. Another reason may be that we use an external dataset to train the model for the ETH dataset.
Finally, we also give some of our detection examples in Figure 6.
5 CONCLUSIONS
In this paper, we propose a novel deep pedestrian detection method based on the Fast R-CNN framework. To reduce the training time and improve the generalization performance, we add the batch
normalization layer between the convolutional layer and activation function layer. To further improve the pedestrian detection speed and accuracy, we use the EdgeBoxes algorithm to remove the redundant windows with poor quality. The experiments show that the proposed method can achieve satisfactory performance comparable with other state-of-the-art methods on two popular pedestrian detection benchmark datasets, INRIA and ETH.
As the image quality can affect the final detection performance, using our proposed method to achieve satisfactory detection performance on low resolution image is still an open issue.
ACKNOWLEDGMENTS
This research was supported by the National Natural Science Foundation of China (No.61672203 & 61375047), the Program for Changjiang Scholars and Innovative Research Team in University of the Ministry of Education of China (No.IRT13059), and the Fundamental Research Funds for the Central Universities of China.

This paper presents experimental results from an EdgeBoxes + Fast R-CNN detector on the task of localizing pedestrians. It uses an AlexNet (CaffeNet) backbone architecture modified to include batch normalization. Experimental results are presented on the INRIA and ETH datasets.
Pros
- The paper is clearly written and easy to follow
Cons
- The paper's two contributions are too minor to merit publication
- Experimental results should include at least the Caltech pedestrian dataset but likely also the KITTI pedestrian dataset
- Recent work from ECCV 2016 [a], with superior results and much more experimental evaluation, is not cited or discussed
My rating is due primarily to the lack luster contributions. The first claimed contribution is the use of EdgeBoxes as proposals for pedestrian detection. Unless the result of this choice produced a truly surprising experimental result, this is simply too minor to be considered a contribution. Moreover, if this choice is important, then the paper should justify it by showing that other proposal methods (of which there are a great many in addition to Selective Search and Edge Boxes) are worse performing in some regard (speed, accuracy, memory, etc.). The second claimed contribution is the use of batch normalization (BN) in their network architecture. There is a case to be made that BN hasn't been explored in Fast R-CNN. However, if the goal of the paper was to thoroughly explore BN + Fast R-CNN, then why focus narrowly on pedestrian detection? Instead, it should focus more broadly on generic object category detection for which there are well established Fast R-CNN baselines on PASCAL VOC and COCO. The use of BN + Fast R-CNN only for pedestrian detection does not provide much signal about this choice. There are also potential technical issues that are not discussed. BN is typically avoided in Fast R-CNN because the batch size seen by most of the network is usually only one or two images. This is likely too few images for the naive application of BN.
[a] "Is Faster R-CNN Doing Well for Pedestrian Detection?" Zhang et al.


This paper introduces a new large-scale music dataset, MusicNet, to serve as a source of supervision and evaluation of machine learning methods for music research. MusicNet consists of hundreds of freely-licensed classical music recordings by 10 composers, written for 11 instruments, together with instrument/note annotations resulting in over 1 million temporal labels on 34 hours of chamber music performances under various studio and microphone conditions. The paper defines a multi-label classification task to predict notes in musical recordings, along with an evaluation protocol, and benchmarks several machine learning architectures for this task: i) learning from spectrogram features; ii) endto-end learning with a neural net; iii) end-to-end learning with a convolutional neural net. These experiments show that end-to-end models trained for note prediction learn frequency selective filters as a low-level representation of audio.
1 INTRODUCTION
Music research has benefited recently from the effectiveness of machine learning methods on a wide range of problems from music recommendation (van den Oord et al., 2013; McFee & Lanckriet, 2011) to music generation (Hadjeres & Pachet, 2016); see also the recent demos of the Google Magenta project1. As of today, there is no large publicly available labeled dataset for the simple yet challenging task of note prediction for classical music. The MIREX MultiF0 Development Set (Benetos & Dixon, 2011) and the Bach10 dataset (Duan et al., 2011) together contain less than 7 minutes of labeled music. These datasets were designed for method evaluation, not for training supervised learning methods.
This situation stands in contrast to other application domains of machine learning. For instance, in computer vision large labeled datasets such as ImageNet (Russakovsky et al., 2015) are fruitfully used to train end-to-end learning architectures. Learned feature representations have outperformed traditional hand-crafted low-level visual features and lead to tremendous progress for image classification. In (Humphrey et al., 2012), Humphrey, Bello, and LeCun issued a call to action: “Deep architectures often require a large amount of labeled data for supervised training, a luxury music informatics has never really enjoyed. Given the proven success of supervised methods, MIR would likely benefit a good deal from a concentrated effort in the curation of sharable data in a sustainable manner.”
This paper introduces a new large labeled dataset, MusicNet, which is publicly available2 as a resource for learning feature representations of music. MusicNet is a corpus of aligned labels on freely-licensed classical music recordings, made possible by licensing initiatives of the European Archive, the Isabella Stewart Gardner Museum, Musopen, and various individual artists. The dataset consists of 34 hours of human-verified aligned recordings, containing a total of 1, 299, 329 individual labels on segments of these recordings. Table 1 summarizes statistics of MusicNet.
The focus of this paper’s experiments is to learn low-level features of music from raw audio data. In Sect. 4, we will construct a multi-label classification task to predict notes in musical recordings,
1https://magenta.tensorflow.org/ 2http://homes.cs.washington.edu/˜thickstn/musicnet.html.
MusicNet
along with an evaluation protocol. We will consider a variety of machine learning architectures for this task: i) learning from spectrogram features; ii) end-to-end learning with a neural net; iii) endto-end learning with a convolutional neural net. Each of the proposed end-to-end models learns a set of frequency selective filters as low-level features of musical audio, which are similar in spirit to a spectrogram. The learned low-level features are visualized in Figure 1. The learned features modestly outperform spectrogram features; we will explore possible reasons for this in Sect. 5.
2 MUSICNET
Related Works. The experiments in this paper suggest that large amounts of data are necessary to recovering useful features from music; see Sect. 4.5 for details. The Lakh dataset, released this summer based on the work of Raffel & Ellis (2015), offers note-level annotations for many 30- second clips of pop music in the Million Song Dataset (McFee et al., 2012). The syncRWC dataset is a subset of the RWC dataset (Goto et al., 2003) consisting of 61 recordings aligned to scores using the protocol described in Ewert et al. (2009). The MAPS dataset (Emiya et al., 2010) is a mixture of acoustic and synthesized data, which expressive models could overfit. The Mazurka project3 consists of commercial music. Access to the RWC and Mazurka datasets comes at both a cost and inconvenience. Both the MAPS and Mazurka datasets are comprised entirely of piano music.
The MusicNet Dataset. MusicNet is a public collection of labels (exemplified in Table 2) for 330 freely-licensed classical music recordings of a variety of instruments arranged in small chamber ensembles under various studio and microphone conditions. The recordings average 6 minutes in length. The shortest recording in the dataset is 55 seconds and the longest is almost 18 minutes. Table 1 summarizes the statistics of MusicNet with breakdowns into various types of labels. Table 2 demonstrates examples of labels from the MusicNet dataset.
MusicNet labels come from 513 label classes using the most naive definition of a class: distinct instrument/note combinations. The breakdowns reported in Table 1 indicate the number of distinct notes that appear for each instrument in our dataset. For example, while a piano has 88 keys only 83 of them are performed in MusicNet. For many tasks a note’s value will be a part of its label, in which case the number of classes will expand by approximately an order of magnitude after taking the cartesian product of the set of classes with the set of values: quarter-note, eighth-note, triplet, etc. Labels regularly overlap in the time series, creating polyphonic multi-labels.
MusicNet is skewed towards Beethoven, thanks to the composer’s popularity among performing ensembles. The dataset is also skewed towards Solo Piano due to an abundance of digital scores available for piano works. For training purposes, researchers may want to augment this dataset to increase coverage of instruments such as Flute and Oboe that are under-represented in MusicNet. Commercial recordings could be used for this purpose and labeled using the alignment protocol described in Sect. 3.
3 DATASET CONSTRUCTION
MusicNet recordings are freely-licensed classical music collected from the European Archive, the Isabella Stewart Gardner Museum, Musopen, and various artists’ collections. The MusicNet labels are retrieved from digital MIDI scores, collected from various archives including the Classical Archives (classicalarchives.com) Suzuchan’s Classic MIDI (suzumidi.com) and HarfeSoft (harfesoft.de). The methods in this section produce an alignment between a digital score and a corresponding freely-licensed recording. A recording is labeled with events in the score, associated to times in the performance via the alignment. Scores containing 6, 550, 760 additional labels are available on request to researchers who wish to augment MusicNet with commercial recordings.
Music-to-score alignment is a long-standing problem in the music research and signal processing communities (Raphael, 1999). Dynamic time warping (DTW) is a classical approach to this problem. An early use of DTW for music alignment is Orio & Schwarz (2001) where a recording is
3http://www.mazurka.org.uk/
aligned to a crude synthesis of its score, designed to capture some of the structure of an overtone series. The method described in this paper aligns recordings to synthesized performances of scores, using side information from a commercial synthesizer. To the best of our knowledge, commercial synthesis was first used for the purpose of alignment in Turetsky & Ellis (2003).
The majority of previous work on alignment focuses on pop music. This is more challenging than aligning classical music because commercial synthesizers do a poor job reproducing the wide variety of vocal and instrumental timbers that appear in modern pop. Furthermore, pop features inharmonic instruments such as drums for which natural metrics on frequency representations–including `2–are not meaningful. For classical music to score alignment, a variant of the techniques described in Turetsky & Ellis (2003) works robustly. This method is described below; we discuss the evaluation of this procedure and its error rate on MusicNet in the appendix.
In order to align the performance with a score, we need to define a metric that compares short segments of the score with segments of a performance. Musical scores can be expressed as binary vectors in E × K where E = {1, . . . , n} and K is a dictionary of notes. Performances reside in RT×p, where T ∈ {1, . . . ,m} is a sequence of time steps and p is the dimensionality of the spectrogram at time T . Given some local cost function C : (Rp,K)→ R, a score Y ∈ E ×K, and a performance X ∈ RT×p, the alignment problem is to
minimize t∈Zn n∑ i=1 C(Xti ,Yi) subject to t0 = 0, tn = m, ti ≤ tj if i < j.
(1)
Dynamic time warping gives an exact solution to the problem in O(mn) time and space. The success of dynamic time warping depends on the metric used to compare the score and the performance. Previous works can be broadly categorized into three groups that define an alignment cost C between segments of music x and score y by injecting them into a common normed space via maps Ψ and Φ:
C(x, y) = ‖Ψ(x)− Φ(y)‖ (2)
The most popular approach–and the one adopted by this paper–maps the score into the space of the performance (Orio & Schwarz, 2001; Turetsky & Ellis, 2003; Soulez et al., 2003). An alternative approach maps both the score and performance into some third space, commonly a chromogram space (Hu et al., 2003; Izmirli & Dannenberg, 2010; Joder et al., 2013). Finally, some recent methods consider alignment in score space, taking Φ = Id and learning Ψ (Garreau et al., 2014; Lajugie et al., 2016).
With reference to the general cost (2), we must specify the maps Ψ,Φ, and the norm ‖ · ‖. We compute the cost in the performance feature space Rp, hence we take Ψ = Id. For the features, we use the log-spectrogram with a window size of 2048 samples. We use a stride of 512 samples between features. Hence adjacent feature frames are computed with 75% overlap. For audio sampled at 44.1kHz, this results in a feature representation with 44, 100/512 ≈ 86 frames per second. A discussion of these parameter choices can be found in the appendix. The map Φ is computed by a synthetizer: we used Plogue’s Sforzando sampler together with Garritan’s Personal Orchestra 4 sample library.
For a (pseudo)-metric on Rp, we take the `2 norm ‖ · ‖2 on the low 50 dimensions of Rp. Recall that Rp represents Fourier components, so we can roughly interpret the k’th coordinate of Rp as the energy associated with the frequency k × (22, 050/1024) ≈ k × 22.5Hz, where 22, 050Hz is the Nyquist frequency of a signal sampled at 44.1kHz. The 50 dimension cutoff is chosen empirically: we observe that the resulting alignments are more accurate using a small number of low-frequency bins rather than the full space Rp. Synthesizers do not accurately reproduce the high-frequency features of a musical instrument; by ignoring the high frequencies, we align on a part of the spectrum where the synthesis is most accurate. The proposed choice of cutoff is aggressive compared to usual settings; for instance, Turetsky & Ellis (2003) propose cutoffs in the 2.5kHz range. The fundamental frequencies of many notes in MusicNet are higher than the 50 × 22.5Hz ≈ 1kHz cutoff. Nevertheless, we find that all notes align well using only the low-frequency information.
4 METHODS
We consider identification of notes in a segment of audio x ∈ X as a multi-label classification problem, modeled as follows. Assign each audio segment a binary label vector y ∈ {0, 1}128. The 128 dimensions correspond to frequency codes for notes, and yn = 1 if note n is present at the midpoint of x. Let f : X → H indicate a feature map. We train a multivariate linear regression to predict ŷ given f(x), which we optimize for square loss. The vector ŷ can be interpreted as a multi-label estimate of notes in x by choosing a threshold c and predicting label n iff ŷn > c. We search for the value c that maximizes F1-score on a sampled subset of MusicNet.
4.1 RELATED WORK
Learning on raw audio is studied in both the music and speech communities. Supervised learning on music has been driven by access to labeled datasets. Pop music labeled with chords (Harte, 2010) has lead to a long line of work on chord recognition, most recently Korzeniowsk & Widmer (2016). Genre labels and other metadata has also attracted work on representation learning, for example Dieleman & Schrauwen (2014). There is also substantial work modeling raw audio representations of speech; a current example is Tokuda & Zen (2016). Recent work from Google DeepMind explores generative models of raw audio, applied to both speech and music (van den Oord et al., 2016).
The music community has worked extensively on a closely related problem to note prediction: fundamental frequency estimation. This is the analysis of fundamental (in contrast to overtone) frequencies in short audio segments; these frequencies are typically considered as proxies for notes. Because access to large labeled datasets was historically limited, most of these works are unsupervised. A good overview of this literature can be found in Benetos et al. (2013). Variants of non-negative matrix factorization are popular for this task; a recent example is Khlif & Sethu (2015). A different line of work models audio probabilistically, for example Berg-Kirkpatrick et al. (2014). Recent work by Kelz et al. (2016) explores supervised models, trained using the MAPS piano dataset.
4.2 MULTI-LAYER PERCEPTRONS
We build a two-layer network with features fi(x) = log ( 1 + max(0,wTi x) ) . We find that compression introduced by a logarithm improves performance versus a standard ReLU network (see Table 3). Figure 1 illustrates a selection of weights wi learned by the bottom layer of this network. The weights learned by the network are modulated sinusoids. This explains the effectiveness of spectrograms as a low-level representation of musical audio. The weights decay at the boundaries, analogous to Gabor filters in vision. This behavior is explained by the labeling methodology: the audio segments used here are approximately 1/3 of a second long, and a segment is given a note
label if that note is on in the center of the segment. Therefore information at the boundaries of the segment is less useful for prediction than information nearer to the center.
4.3 (LOG-)SPECTROGRAMS
Spectrograms are an engineered feature representation for musical audio signals, available in popular software packages such as librosa (McFee et al., 2015). Spectrograms (resp. log-spectrograms) are closely related to a two-layer ReLU network (resp. the log-ReLU network described above). If x = (x1, . . . , xt) denotes a segment of an audio signal of length t then we can define
Speck(x) ≡ ∣∣∣∣∣ t−1∑ s=0 e−2πiks/txs ∣∣∣∣∣ 2 = ( t−1∑ s=0 cos(2πks/t)xs )2 + ( t−1∑ s=0 sin(2πks/t)xs )2 .
These features are not precisely learnable by a two-layer ReLU network. But recall that |x| = max(0, x) + max(0,−x) and if we take weight vectors u,v ∈ RT with us = cos(2πks/t) and vs = sin(2πks/t) then the ReLU network can learn
fk,cos(x) + fk,sin(x) ≡ |uTx|+ |vTx| = ∣∣∣∣∣ t−1∑ s=0 cos(2πks/t)xs ∣∣∣∣∣+ ∣∣∣∣∣ t−1∑ s=0 sin(2πks/t)xs ∣∣∣∣∣ . We call this family of features a ReLUgram and observe that it has a similar form to the spectrogram; we merely replace the x 7→ x2 non-linearity of the spectrogram with x 7→ |x|. These features achieve similar performance to spectrograms on the classification task (see Table 3).
4.4 WINDOW SIZE
When we parameterize a network, we must choose the width of the set of weights in the bottom layer. This width is called the receptive field in the vision community; in the music community it is called the window size. Traditional frequency analyses, including spectrograms, are highly sensitive to the window size. Windows must be long enough to capture relevant information, but not so long that they lose temporal resolution; this is the classical time-frequency tradeoff. Furthermore, windowed frequency analysis is subject to boundary effects, known as spectral leakage. Classical signal processing attempts to dampen these effects with predefined window functions, which apply a mask that attenuates the signal at the boundaries (Rabiner & Schafer, 2007).
The proposed end-to-end models learn window functions. If we parameterize these models with a large window size then the model will learn that distant information is irrelevant to local prediction, so the magnitude of the learned weights will attenuate at the boundaries. We therefore focus on two window sizes: 2048 samples, which captures the local content of the signal, and 16,384 samples, which is sufficient to capture almost all relevant context (again see Figure 1).
4.5 REGULARIZATION
The size of MusicNet is essential to achieving the results in Figure 1. In Figure 3 (Left) we optimize a two-layer ReLU network on a small subset of MusicNet consisting of 65, 000 monophonic data points. While these features do exhibit dominant frequencies, the signal is quite noisy. Comparable noisy frequency selective features were recovered by Dieleman & Schrauwen (2014); see their Figure 3. We can recover clean features on a small dataset using heavy regularization, but this destroys classification performance; regularizing with dropout poses a similar tradeoff. By contrast, Figure 3 (Right) shows weights learned by an unregularized two-layer network trained on the full MusicNet dataset. The models described in this paper do not overfit to MusicNet and optimal performance (reported in Table 3) is achieved without regularization.
4.6 CONVOLUTIONAL NETWORKS
Previously, we estimated ŷ by regressing against f(x). We now consider a convolutional model that regresses against features of a collection of shifted segments x` near to the original segment x. The learned features of this network are visually comparable to those learned by the fully connected network (Figure 1). The parameters of this network are the receptive field, stride, and pooling regions.
The results reported in Table 3 are achieved with 500 hidden units using a receptive field of 2, 048 samples with an 8-sample stride across a window of 16, 384 samples. These features are grouped into average pools of width 16, with a stride of 8 features between pools. A max-pooling operation yields similar results. The learned features are consistent across different parameterizations. In all cases the learned features are comparable to those of a fully connected network.
5 RESULTS
We hold out a test set of 3 recordings for all the results reported in this section:
• Bach’s Prelude in D major for Solo Piano. WTK Book 1, No 5. Performed by Kimiko Ishizaka. MusicNet recording id 2303. • Mozart’s Serenade in E-flat major. K375, Movement 4 - Menuetto. Performed by the Soni Ventorum Wind Quintet. MusicNet recording id 1819. • Beethoven’s String Quartet No. 13 in B-flat major. Opus 130, Movement 2 - Presto. Released by the European Archive. MusicNet recording id 2382.
The test set is a representative sampling of MusicNet: it covers most of the instruments in the dataset in small, medium, and large ensembles. The test data points are evenly spaced segments separated by 512 samples, between the 1st and 91st seconds of each recording. For the wider features, there is substantial overlap between adjacent segments. Each segment is labeled with the notes that are on in the middle of the segment.
We evaluate our models on three scores: precision, recall, and average precision. The precision score is the count of correct predictions by the model (across all data points) divided by the total number
of predictions by the model. The recall score is the count of correct predictions by the model divided by the total number of (ground truth) labels in the test set. Precision and recall are parameterized by the note prediction threshold c (see Sect. 4). By varying c, we construct precision-recall curves (see Figure 4). The average precision score is the area under the precision-recall curve.
A spectrogram of length n is computed from 2n samples, so the linear 1024-point spectrogram model is directly comparable to the MLP runs with 2048 raw samples. Learned features4 modestly outperform spectrograms for comparable window sizes. The discussion of windowing in Sect. 4.4 partially explains this. Figure 5 suggests a second reason. Recall (Sect. 4.3) that the spectrogram features can be interpreted as the magnitude of the signal’s inner product with sine waves of linearly spaced frequencies. In contrast, the proposed networks learn weights with frequencies distributed similarly to the distribution of notes in MusicNet (Figure 5). This gives the network higher resolution in the most critical frequency regions.
ACKNOWLEDGMENTS
We thank Bob L. Sturm for his detailed feedback on an earlier version of the paper. We also thank Brian McFee and Colin Raffel for fruitful discussions. Sham Kakade acknowledges funding from the Washington Research Foundation for innovation in Data-intensive Discovery. Zaid Harchaoui acknowledges funding from the program ”Learning in Machines and Brains” of CIFAR.
4A demonstration using learned MLP features to synthesize a musical performance is available on the dataset webpage: http://homes.cs.washington.edu/˜thickstn/demos.html
B ALIGNMENT PARAMETER ROBUSTNESS
The definitions of audio featurization and the alignment cost function were contingent on several parameter choices. These choices were optimized by systematic exploration of the parameter space. We investigated what happens as we vary each parameter and made the choices that gave the best results in our listening tests. Fine-tuning of the parameters yields marginal gains.
The quality of alignments improves uniformly with the quality of synthesis. The time-resolution of labels improves uniformly as the stride parameter decreases; minimization of stride is limited by system memory constraints. We find that the precise phase-invariant feature specification has little effect on alignment quality. We experimented with spectrograms and log-spectrograms using windowed and un-windowed signals. Alignment quality seemed to be largely unaffected.
The other parameters are governed by a tradeoff curve; the optimal choice is determined by balancing desirable outcomes. The Fourier window size is a classic tradeoff between time and frequency resolution. The `2 norm can be understood as a tradeoff between the extremes of `1 and `∞. The `1 norm is too egalitarian: the preponderance of errors due to synthesis quality add up and overwhelm the signal. On the other hand, the `∞ norm ignores too much of the signal in the spectrogram. The spectrogram cutoff, discussed in Sec. 3, is also a tradeoff between synthesis quality and maximal use of information
C ADDITIONAL ERROR ANALYSIS
For each model, using the test set described in Sect. 5, we report accuracy and error scores used by the MIR community to evaluate the Multi-F0 systems. Definitions and a discussion of these metrics are presented in Poliner & Ellis (2007).
D PRECISION & RECALL CURVES
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 7: The 500 node, 2048 raw sample MLP.
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 9: The average pooling model.
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 11: The convolutional model.
E ADDITIONAL RESULTS
We report additional results on splits of the test set described in Sect. 5.

The paper introduces a new dataset called MusicNet (presumably analogous to ImageNet), featuring dense ground truth labels for 30+ hours of classical music, which is provided as raw audio. Such a dataset is extremely valuable for music information retrieval (MIR) research and a dataset of this size has never before been publicly available. It has the potential to dramatically increase the impact of modern machine learning techniques (e.g. deep learning) in this field, whose adoption has previously been hampered by a lack of available datasets that are large enough. The paper is clear and well-written.
The paper also features some "example" experiments using the dataset, which I am somewhat less excited about. The authors decided to focus on one single task that is not particularly challenging: identifying pitches in isolated segments of audio. Pitch information is a fairly low-level characteristic of music. Considering that isolated fragments are used as input, this is a relatively simple problem that probably doesn't even require machine learning to solve adequately, e.g. peak picking on a spectral representation could already get you pretty far. It's not clear what value the machine learning component in the proposed approach actually adds, if any. I could be wrong about this as I haven't done the comparison myself, but I think the burden is on the authors to demonstrate that using ML here is actually useful.
I would argue that one of the strenghts of the dataset is the variety of label information it provides, so a much more convincing setup would have been to demonstrate many different prediction tasks for both low-level (e.g. pitch, onsets) and high-level (e.g. composer) characteristics, perhaps with fewer and simpler models -- maybe even sticking to spectrogram input and forgoing raw audio input for the time being, as this comparison seems orthogonal to the introduction of the dataset itself. As it stands, I feel that the fact that the experiments are relatively uninteresting detracts from the main point of the paper, which is to introduce a new public dataset that is truly unique in terms of its scale and scope.
That said, the experiments seem to have been conducted in a rigorous fashion and the evaluation and analysis of the resulting models is properly executed.
Re: Section 4.5, it is rather unsurprising to me that a pitch detector would learn filters that resemble pitches (i.e. sinusoids), although the observation that this requires a relatively large amount of data is interesting. However, it would be more interesting to demonstrate that this is also the case for higher-level tasks. The authors favourably compare the features learnt by their model with prior work on end-to-end learning from raw audio, but neglect that the tasks considered in this work were much more high-level.
Some might also question whether ICLR is the appropriate venue to introduce a new dataset, but personally I think it's a great idea to submit it here, seeing as it will reach the right people. I suppose this is up to the organisers and the program committee, but I thought it important to mention this, because I don't think this paper merits acceptance based on its experimental results alone.


MULTI-VIEW RECURRENT NEURAL ACOUSTIC WORD EMBEDDINGS
Recent work has begun exploring neural acoustic word embeddings—fixeddimensional vector representations of arbitrary-length speech segments corresponding to words. Such embeddings are applicable to speech retrieval and recognition tasks, where reasoning about whole words may make it possible to avoid ambiguous sub-word representations. The main idea is to map acoustic sequences to fixed-dimensional vectors such that examples of the same word are mapped to similar vectors, while different-word examples are mapped to very different vectors. In this work we take a multi-view approach to learning acoustic word embeddings, in which we jointly learn to embed acoustic sequences and their corresponding character sequences. We use deep bidirectional LSTM embedding models and multi-view contrastive losses. We study the effect of different loss variants, including fixed-margin and cost-sensitive losses. Our acoustic word embeddings improve over previous approaches for the task of word discrimination. We also present results on other tasks that are enabled by the multi-view approach, including cross-view word discrimination and word similarity.
1 INTRODUCTION
Word embeddings—continuous-valued vector representations of words—are an almost ubiquitous component of recent natural language processing (NLP) research. Word embeddings can be learned using spectral methods (Deerwester et al., 1990) or, more commonly in recent work, via neural networks (Bengio et al., 2003; Mnih & Hinton, 2007; Mikolov et al., 2013; Pennington et al., 2014). Word embeddings can also be composed to form embeddings of phrases, sentences, or documents (Socher et al., 2014; Kiros et al., 2015; Wieting et al., 2016; Iyyer et al., 2015).
In typical NLP applications, such embeddings are intended to represent the semantics of the corresponding words/sequences. In contrast, embeddings that represent the way a word or sequence sounds are rarely considered. In this work we address this problem, starting with embeddings of individual words. Such embeddings could be useful for tasks like spoken term detection (Fiscus et al., 2007), spoken query-by-example search (Anguera et al., 2014), or even speech recognition using a whole-word approach (Gemmeke et al., 2011; Bengio & Heigold, 2014). In tasks that involve comparing speech segments to each other, vector embeddings can allow more efficient and more accurate distance computation than sequence-based approaches such as dynamic time warping (Levin et al., 2013, 2015; Kamper et al., 2016; Settle & Livescu, 2016; Chung et al., 2016).
We consider the problem of learning vector representations of acoustic sequences and orthographic (character) sequences corresponding to single words, such that the learned embeddings represent the way the word sounds. We take a multi-view approach, where we jointly learn the embeddings for character and acoustic sequences. We consider several contrastive losses, based on learning from pairs of matched acoustic-orthographic examples and randomly drawn mismatched pairs. The losses correspond to different goals for learning such embeddings; for example, we might want the embeddings of two waveforms to be close when they correspond to the same word and far when they correspond to different ones, or we might want the distances between embeddings to correspond to some ground-truth orthographic edit distance.
One of the useful properties of this multi-view approach is that, unlike earlier work on acoustic word embeddings, it produces both acoustic and orthographic embeddings that can be directly compared. This makes it possible to use the same learned embeddings for multiple single-view and cross-view tasks. Our multi-view embeddings produce improved results over earlier work on acoustic word discrimination, as well as encouraging results on cross-view discrimination and word similarity.1
2 OUR APPROACH
In this section, we first introduce our approach for learning acoustic word embeddings in a multiview setting, after briefly reviewing related approaches to put ours in context. We then discuss the particular neural network architecture we use, based on bidirectional long short-term memory (LSTM) networks (Hochreiter & Schmidhuber, 1997).
2.1 MULTI-VIEW LEARNING OF ACOUSTIC WORD EMBEDDINGS
Previous approaches have focused on learning acoustic word embeddings in a “single-view” setting. In the simplest approach, one uses supervision of the form “acoustic segment x is an instance of the word y”, and trains the embedding to be discriminative of the word identity. Formally, given a dataset of paired acoustic segments and word labels {(xi, yi)}Ni=1, this approach solves the following optimization:
min f,h
objclassify := 1
N N∑ i ` (h(f(xi)),yi) , (1)
where network f maps an acoustic segment into a fixed-dimensional feature vector/embedding, h is a classifier that predicts the corresponding word label from the label set of the training data, and the loss ` measures the discrepancy between the prediction and ground-truth word label (one can use any multi-class classification loss here, and a typical choice is the cross-entropy loss where h has a softmax top layer). The two networks f and h are trained jointly. Equivalently, one could consider the composition h(f(x)) as a classifier network, and use any intermediate layer’s activations as the features. We refer to the objective in (1) as the “classifier network” objective, which has been used in several prior studies on acoustic word embeddings (Bengio & Heigold, 2014; Kamper et al., 2016; Settle & Livescu, 2016).
This objective, however, is not ideal for learning acoustic word embeddings. This is because the set of possible word labels is huge, and we may not have enough instances of each label to train a good classifier. In downstream tasks, we may encounter acoustic segments of words that did not appear in the embedding training set, and it is not clear that the classifier-based embeddings will have reasonable behavior on previously unseen words.
An alternative approach, based on Siamese networks (Bromley et al., 1993), uses supervision of the form “segment x1 is similar to segment x2, and is not similar to segment x3”, where two segments are considered similar if they have the same word label and dissimilar otherwise. Models based on Siamese networks have been used for a variety of representation learning problems in NLP (Hu et al., 2014; Wieting et al., 2016), vision (Hadsell et al., 2006), and speech (Synnaeve et al., 2014; Kamper et al., 2015) including acoustic word embeddings (Kamper et al., 2016; Settle & Livescu, 2016). A typical objective in this category enforces that the distance between (x1, x3) is larger than the distance between (x1, x2) by some margin:
min f
objsiamese := 1
N N∑ i max ( 0, m+ dis ( f(x1i ), f(x 2 i ) ) − dis ( f(x1i ), f(x 3 i ) )) , (2)
where the network f extracts the fixed-dimensional embedding, the distance function dis (·, ·) measures the distance between the two embedding vectors, andm > 0 is the margin parameter. The term “Siamese” (Bromley et al., 1993; Chopra et al., 2005) refers to the fact that the triplet (x1, x2, x3) share the same embedding network f .
Unlike the classification-based loss, the Siamese network loss does not enforce hard decisions on the label of each segment. Instead it tries to learn embeddings that respect distances between word
1Our tensorflow implementation is available at https://github.com/opheadacheh/Multi-view-neural-acoustic-words-embeddings
pairs, which can be helpful for dealing with unseen words. The Siamese network approach also uses more examples in training, as one can easily generate many more triplets than (segment, label) pairs, and it is not limited to those labels that occur a sufficient number of times in the training set.
The above approaches treat the word labels as discrete classes, which ignores the similarity between different words, and does not take advantage of the more complex information contained in the character sequences corresponding to word labels. The orthography naturally reflects some aspects of similarity between the words’ pronunciations, which should also be reflected in the acoustic embeddings. One way to learn features from multiple sources of complementary information is using a multi-view representation learning setting. We take this approach, and consider the acoustic segment and the character sequence to be two different views of the pronunciation of the word.
While many deep multi-view learning objectives are applicable (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2014; Sohn et al., 2014; Wang et al., 2015), we consider the multi-view contrastive loss objective of (Hermann & Blunsom, 2014), which is simple to optimize and implement and performs well in practice. In this algorithm, we embed acoustic segments x by a network f and character label sequences c by another network g into a common space, and use weak supervision of the form “for paired segment x+ and its character label sequence c+, the distance between their embedding is much smaller than the distance between embeddings of x+ and an unmatched character label sequence c−”. Formally, we optimize the following objective with such supervision:
min f,g
obj0 := 1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(c + i ) ) − dis ( f(x+i ), g(c − i ) )) , (3)
where c−i is a negative character label sequence of x + i to be contrasted with the positive/correct character sequence c+i , and m is the margin parameter. In this paper we use the cosine distance,
dis (a,b) = 1− 〈
a ‖a‖ , b ‖b‖
〉 .2
Note that in the multi-view setting, we have multiple ways of generating triplets that contain one positive pair and one negative pair each. Below are the other three objectives we explore in this paper:
min f,g
obj1 := 1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(c + i ) ) − dis ( g(c+i ), g(c − i ) )) , (4)
min f,g
obj2 := 1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(c + i ) ) − dis ( f(x−i ), g(c + i ) )) , (5)
min f,g
obj3 := 1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(c + i ) ) − dis ( f(x+i ), f(x − i ) )) . (6)
x−i in (5) and (6) refers to a negative acoustic feature sequence, that is one with a different label from x+i . We note that obj
1 and obj3 contain distances between same-view embeddings, and are less thoroughly explored in the literature. We will also consider combinations of obj0 through obj3.
Finally, thus far we have considered losses that do not explicitly take into account the degree of difference between the positive and negative pairs (although the learned embeddings may implicitly learn this through the relationship between sequences in the two views). We also consider a costsensitive objective designed to explicitly arrange the embedding space such that word similarity is respected. In (3), instead of a fixed margin m, we use:
m(c+, c−) := mmax · min (tmax, editdis(c
+, c−))
tmax , (7)
where tmax > 0 is a threshold for edit distances (all edit distances above tmax are considered equally bad), and mmax is the maximum margin we impose. The margin is set to mmax if the edit distance between two character sequences is above tmax; otherwise it scales linearly with the edit distance editdis(c+, c−)). We use the Levenshtein distance as the edit distance. Here we explore the costsensitive margin with obj0, but it could in principle be used with other objectives as well.
2In experiments, we use the unit-length vector a‖a‖ as the embedding. It tends to perform better than f(x) and more directly reflects the cosine similarity. This is equivalent to adding a nonlinear normalization layer on top of f .
2.2 RECURRENT NEURAL NETWORK ARCHITECTURE
Since the inputs of both views have a sequential structure, we implement both f and g with recurrent neural networks and in particular long-short term memory networks (LSTMs). Recurrent neural networks are the state-of-the-art models for a number of speech tasks including speech recognition Graves et al. (2013), and LSTM-based acoustic word embeddings have produced the best results on one of the tasks in our experiments (Settle & Livescu, 2016).
As shown in Figure 1, our f and g are produced by multi-layer (stacked) bidirectional LSTMs. The inputs can be any frame-level acoustic feature representation and vector representation of the characters in the orthographic input. At each layer, two LSTM cells process the input sequence from left to right and from right to left respectively. At intermediate layers, the outputs of the two LSTMs at each time step are concatenated to form the input sequence to the next layer. At the top layer, the last time step outputs of the two LSTMs are concatenated to form a fixed-dimensional embedding of the view, and the embeddings are then used to calculate the cosine distances in our objectives.
3 RELATED WORK
We are aware of no prior work on multi-view learning of acoustic and character-based word embeddings. However, acoustic word embeddings learned in other ways have recently begun to be studied. Levin et al. (2013) proposed an approach for embedding an arbitrary-length segment of speech as a fixed-dimensional vector, based on representing each word as a vector of dynamic time warping (DTW) distances to a set of template words. This approach produced improved performance on a word discrimination task compared to using raw DTW distances, and was later also applied successfully for a query-by-example task (Levin et al., 2015). One disadvantage of this approach is that, while DTW handles the issue of variable sequence lengths, it is computationally costly and involves a number of DTW parameters that are not learned.
Kamper et al. (2016) and Settle & Livescu (2016) later improved on Levin et al.’s word discrimination results using convolutional neural networks (CNNs) and recurrent neural networks (RNNs) trained with either a classification or contrastive loss. Bengio & Heigold (2014) trained convolutional neural network (CNN)-based acoustic word embeddings for rescoring the outputs of a speech recognizer, using a loss combining classification and ranking criteria. Maas et al. (2012) trained a CNN to predict a semantic word embedding from an acoustic segment, and used the resulting embeddings as features in a segmental word-level speech recognizer. Harwath and Glass Harwath & Glass (2015); Harwath et al. (2016); Harwath & Glass (2017) jointly trained CNN embeddings of images and spoken captions, and showed that word-like unit embeddings can be extracted from the speech model. CNNs require normalizing the duration of the input sequences, which has typically been done via padding. RNNs, on the other hand, are more flexible in dealing with very different-length sequences. Chen et al. (2015) used long short-term memory (LSTM) networks with a classification loss to embed acoustic words for a simple (single-query) query-by-example search task. Chung et al. (2016) learned acoustic word embeddings based on recurrent neural network (RNN) autoencoders, and found that they improve over DTW for a word discrimination task similar to that of Levin et al. (2013). Audhkhasi et al. (2017) learned autoencoders for acoustic and written words, as well as a model for comparing the two, and applied these to a keyword search task.
Evaluation of acoustic word embeddings in downstream tasks such as speech recognition and search can be costly, and can obscure details of embedding models and training approaches. Most evaluations have been based on word discrimination – the task of determining whether two speech segments correspond to the same word or not – which can be seen as a proxy for query-by-example search (Levin et al., 2013; Kamper et al., 2016; Settle & Livescu, 2016; Chung et al., 2016). One difference between word discrimination and search/recognition tasks is that in word discrimination the word boundaries are given. However, prior work has been able to apply results from word discrimination Levin et al. (2013) to improve a query-by-example system without known word boundaries Levin et al. (2015), by simply applying their embeddings to non-word segments as well.
The only prior work focused on vector embeddings of character sequences explicitly aimed at representing their acoustic similarity is that of Ghannay et al. (2016), who proposed evaluations based on nearest-neighbor retrieval, phonetic/orthographic similarity measures, and homophone disambiguation. We use related tasks here, as well as acoustic word discrimination for comparison with prior work on acoustic embeddings.
4 EXPERIMENTS AND RESULTS
The ultimate goal is to gain improvements in speech systems where word-level discrimination is needed, such as speech recognition and query-by-example search. However, in order to focus on the content of the embeddings themselves and to more quickly compare a variety of models, it is desirable to have surrogate tasks that serve as intrinsic measures of performance. Here we consider three forms of evaluation, all based on measuring whether cosine distances between learned embeddings correspond well to desired properties.
In the first task, acoustic word discrimination, we are given a pair of acoustic sequences and must decide whether they correspond to the same word or to different words. This task has been used in several prior papers on acoustic word embeddings Kamper et al. (2015, 2016); Chung et al. (2016); Settle & Livescu (2016) and is a proxy for query-by-example search. For each given spoken word pair, we calculate the cosine distance between their embeddings. If the cosine distance is below a threshold, we output “yes” (same word), otherwise we output “no” (different words). The performance measure is the average precision (AP), which is the area under the precision-recall curve generated by varying the threshold and has a maximum value of 1.
In our multi-view setup, we embed not only the acoustic words but also the character sequences. This allows us to use our embeddings also for tasks involving comparisons between written and spoken words. For example, the standard task of spoken term detection (Fiscus et al., 2007) involves searching for examples of a given text query in spoken documents. This task is identical to queryby-example except that the query is given as text. In order to explore the potential of multi-view embeddings for such tasks, we design another proxy task, cross-view word discrimination. Here we are given a pair of inputs, one a written word and one an acoustic word segment, and our task is to determine if the acoustic signal is an example of the written word. The evalution proceeds analogously to the acoustic word discrimination task: We output “yes” if the cosine distance between the embeddings of the written and spoken sequences are below some threshold, and measure performance as the average precision (AP) over all thresholds.
Finally, we also would like to obtain a more fine-grained measure of whether the learned embeddings capture our intuitive sense of similarity between words. Being able to capture word similarity may also be useful in building query or recognition systems that fail gracefully and produce humanlike errors. For this purpose we measure the rank correlation between embedding distances and character edit distances. This is analogous to the evaluation of semantic word embeddings via the rank correlation between embedding distances and human similarity judgments (Finkelstein et al., 2001; Hill et al., 2015). In our case, however, we do not use human judgments since the ground-truth edit distances themselves provide a good measure. We refer to this as the word similarity task, and we apply this measure to both pairs of acoustic embeddings and pairs of character sequence embeddings. Similar measures have been proposed by Ghannay et al. (2016) to evaluate acoustic word embeddings, although they considered only near neighbors of each word whereas we consider the correlation across the full range of word pairs.
In the experiments described below, we first focus on the acoustic word discrimination task for purposes of initial exploration and hyperparameter search, and then largely fix the models for evaluation using the cross-view word discrimination and word similarity measures.
4.1 DATA
We use the same experimental setup and data as in Kamper et al. (2015, 2016); Settle & Livescu (2016). The task and setup were first developed by (Carlin et al., 2011). The data is drawn from the Switchboard English conversational speech corpus (Godfrey et al., 1992). The spoken word segments range in duration from 50 to 200 frames (0.5 - 2 seconds). The train/dev/test splits contain 9971/10966/11024 pairs of acoustic segments and character sequences, corresponding to 1687/3918/3390 unique words. In computing the AP for the dev or test set, all pairs in the set are used, yielding approximately 60 million word pairs.
The input to the embedding model in the acoustic view is a sequence of 39-dimensional vectors (one per frame) of standard mel frequency cepstral coefficients (MFCCs) and their first and second derivatives. The input to the character sequence embedding model is a sequence of 26-dimensional one-hot vectors indicating each character of the word’s orthography.
4.2 MODEL DETAILS AND HYPERPARAMETER TUNING
We experiment with different neural network architectures for each view, varying the number of stacked LSTM layers, the number of hidden units for each layer, and the use of single- or bidirectional LSTM cells. A coarse grid search shows that 2-layer bidirectional LSTMs with 512 hidden units per direction per layer perform well on the acoustic word discrimination task, and we keep this structure fixed for subsequent experiments (see Appendix A for more details). We use the outputs of the top-layer LSTMs as the learned embedding for each view, which is 1024-dimensional if bidirectional LSTMs are used.
In training, we use dropout on the inputs of the acoustic view and between stacked layers for both views. The architecture is illustrated in Figure 1. For each training example, our contrastive losses require a corresponding negative example. We generate a negative character label sequence by uniformly sampling a word label from the training set that is different from the positive label. We perform a new negative label sampling at the beginning of each epoch. Similarly, negative acoustic feature sequences are uniformly sampled from all of the differently labeled acoustic feature sequences in the training set.
The network weights are initialized with values sampled uniformly from the range [−0.05, 0.05]. We use the Adam optimizer (Kingma & Ba, 2015) for updating the weights using mini-batches of 20 acoustic segments, with an initial learning rate tuned over {0.0001, 0.001}. Dropout is used at each layer, with the rate tuned over {0, 0.2, 0.4, 0.5}, in which 0.4 usually outperformed others. The margin in our basic contrastive objectives 0-3 is tuned over {0.3, 0.4, 0.5, 0.6, 0.7}, out of which 0.4 and 0.5 typically yield best results. For obj0 with the cost-sensitive margin, we tune the maximum margin mmax over {0.5, 0.6, 0.7} and the threshold tmax over {9, 11, 13}. We train each model for up to 1000 epochs. The model that gives the best AP on the development set is used for evaluation on the test set.
4.3 EFFECTS OF DIFFERENT OBJECTIVES
We presented four contrastive losses (3)–(6) and potential combinations in Section 2.1. We now explore the effects of these different objectives on the word discrimination tasks.
Table 1 shows the development set AP for acoustic and cross-view word discrimination achieved using the various objectives. We tuned the objectives for the acoustic discrimination task, and then used the corresponding converged models for the cross-view task. Of the simple contrastive objectives, obj0 and obj2 (which involve only cross-view distances) slightly outperform the other two on the acoustic word discrimination task. The best-performing objective is the “symmetrized” objective obj0 + obj2, which significantly outperforms all individual objectives (and the combination of the four). Finally, the cost-sensitive objective is very competitive as well, while falling slightly short of the best performance. We note that a similar objective to our obj0 + obj2 was used by Vendrov et al. (2016) for the task of caption-image retrieval, where the authors essentially use all non-paired
Objective Dev AP Dev AP (acoustic) (cross-view)
obj0 0.659 0.791 obj1 0.654 0.807 obj2 0.675 0.788 obj3 0.640 0.782 obj0 + obj2 0.702 0.814∑3
i=0 obj i 0.672 0.804 cost-sensitive 0.671 0.802
Table 1: Word discrimination performance with different objectives.
examples from the other view in the minibatch as negative examples (instead of random sampling one negative example as we do) to be contrasted with one paired example.
Figure 2 shows the progression of the development set AP for acoustic word discrimination over 1000 training epochs, using several of the objectives, where AP is evaluated every 5 epochs. We observe that even after 1000 epochs, the development set AP has not quite saturated, indicating that it may be possible to further improve performance.
Overall, our best-performing objective is the combined obj0+obj2, and we use it for reporting final test-set results. Table 2 shows the test set AP for both the acoustic and cross-view tasks using our final model (“multi-view LSTM”). For comparison, we also include acoustic word discrimination results reported previously by Kamper et al. (2016); Settle & Livescu (2016). Previous approaches have not addressed the problem of learning embeddings jointly with the text view, so they can not be evaluated on the cross-view task.
4.4 WORD SIMILARITY TASKS
Table 3 gives our results on the word similarity tasks, that is the rank correlation (Spearman’s ρ) between embedding distances and orthographic edit distance (Levenshtein distance between character sequences). We measure this correlation for both our acoustic word embeddings and for our text embeddings. In the case of the text embeddings, we could of course directly measure the Levenshtein distance between the inputs; here we are simply measuring how much of this information the text embeddings are able to retain.
Interestingly, while the cost-sensitive objective did not produce substantial gains on the word discrimination tasks above, it does greatly improve the performance on this word similarity measure. This is a satisfying observation, since the cost-sensitive loss is trying to improve precisely this relationship between distances in the embedding space and the orthographic edit distance.
Although we have trained our embeddings using orthographic labels, it is also interesting to consider how closely aligned the embeddings are with the corresponding phonetic pronunciations. For comparison, the rank correlation between our acoustic embeddings and phonetic edit distances is 0.226, and for our text embeddings it is 0.241, which are relatively close to the rank correlations with orthographic edit distance. A future direction is to directly train embeddings with phonetic sequence supervision rather than orthography; this setting involves somewhat stronger supervision, but it is easy to obtain in many cases.
Another interesting point is that the performance is not a great deal better for the text embeddings than for the acoustic embeddings, even though the text embeddings have at their disposal the text input itself. We believe this has to do with the distribution of words in our data: While the data includes a large variety of words, it does not include many very similar pairs. In fact, of all possible pairs of unique training set words, fewer than 2% have an edit distance below 5 characters. Therefore, there may not be sufficient information to learn to distinguish detailed differences among character sequences, and the cost-sensitive loss ultimately does not learn much more than to separate different words. In future work it would be interesting to experiment with data sets that have a larger variety of similar words.
4.5 VISUALIZATION OF LEARNED EMBEDDINGS
Figure 3 gives a 2-dimensional t-SNE (van der Maaten & Hinton, 2008) visualization of selected acoustic and character sequences from the development set, including some that were seen in the training set and some previously unseen words. The previously seen words in this figure were selected uniformly at random among those that appear at least 15 times in the development set (the unseen words are the only six that appear at least 15 times in the development set). This visualization demonstrates that the acoustic embeddings cluster very tightly and are very close to the text embeddings, and that unseen words cluster nearly as well as previously seen ones.
While Figure 3 shows the relationship among the multiple acoustic embeddings and the text embeddings, the words are all very different so we cannot draw conclusions about the relationships between words. Figure 4 provides another visualization, this time exploring the relationship among the text embeddings of a number of closely related words, namely all development set words ending in “-ly”, “-ing”, and “-tion”. This visualization confirms that related words are embedded close together, with the words sharing a suffix forming fairly well-defined clusters.
5 CONCLUSION
We have presented an approach for jointly learning acoustic word embeddings and their orthographic counterparts. This multi-view approach produces improved acoustic word embedding performance over previous approaches, and also has the benefit that the same embeddings can be applied for both spoken and written query tasks. We have explored a variety of contrastive objectives: ones with a fixed margin that aim to separate same and different word pairs, as well as a cost-sensitive loss that aims to capture orthographic edit distances. While the losses generally perform similarly for word discrimination tasks, the cost-sensitive loss improves the correlation between embedding distances and orthographic distances. One interesting direction for future work is to directly use knowledge about phonetic pronunciations, in both evaluation and training. Another direction is to extend our approach to directly train on both word and non-word segments.
ACKNOWLEDGMENTS
This research was supported by a Google Faculty Award and by NSF grant IIS-1321015. The opinions expressed in this work are those of the authors and do not necessarily reflect the views of the funding agency. This research used GPUs donated by NVIDIA Corporation. We thank Herman Kamper and Shane Settle for their assistance with the data and experimental setup.
A ADDITIONAL ANALYSIS
We first explore the effect of network architectures for our embedding models. We learn embeddings using objective obj0 and evaluate them on the acoustic and cross-view word discrimination tasks. The resulting average precisions on the development set are given in Table 4. All of the models were trained for 1000 epochs, except for the 1-layer unidirectional models which converged after 500 epochs. It is clear that bidirectional LSTMs are more successful than unidirectional LSTMs for these tasks, and two layers of LSTMs are much better than a single layer of LSTMs. We did not observe significant further improvement by using more than two layers of LSTMs. For all other experiments, we fix the architecture to 2-layer bidirectional LSTMs for each view.
In Figure 5 we also give the precision-recall curve for our best models, as well as the scatter plot of cosine distances between acoustic embeddings vs. orthographic edit distances.

This paper proposes an approach to learning word vector representations for character sequences and acoustic spans jointly. The paper is clearly written and both the approach and experiments seem reasonable in terms of execution. The motivation and tasks feel a bit synthetic as it requires acoustics spans for words that have already been segmented from continuous speech - - a major assumption. The evaluation tasks feel a bit synthetic overall and in particular when evaluating character based comparisons it seems there should also be phoneme based comparisons.
There's a lot of discussion of character edit distance relative to acoustic span similarity. It seems very natural to also include phoneme string edit distance in this discussion and experiments. This is especially true of the word similarity test. Rather than only looking at levenshtein edit distance of characters you should evaluate edit distance of the phone strings relative to the acoustic embedding distances. Beyond the evaluation task the paper would be more interesting if you compared character embeddings with phone string embeddings. I believe the last function could remain identical it's just swapping out characters for phones as the symbol set.  finally in this topic the discussion and experiments should look at homophones As if not obvious what the network would learn to handle these.
 the vocabulary size and training data amount make this really a toy problem. although there are many pairs constructed most of those pairs will be easy distinctions. the experiments and conclusions would be far stronger with a larger vocabulary and word segment data set with subsampling all pairs perhaps biased towards more difficult or similar pairs.
 it seems this approach is unable to address the task of keyword spotting in longer spoken utterances. If that's the case please add some discussion as to why you are solving the problem of word embeddings given existing word segmentations. The motivating example of using this approach to retrieve words seems flawed if a recognizer must be used to segment words beforehand