SoundNet: Learning Sound Representations from Unlabeled Video
We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.
This paper propose to use pre-trained CNNs for visual recognition to be used as a teacher to the learn a student sound network on the 2M acoustic unlabelled videos taken from fliker to learn natural sound representations.  The idea of transferring abstract knowledge learned from one modality to other in the student teacher network is interesting and looks appealing. Although the methodology used is not very novel it facilitate learning of much deeper sound-net. The convolution features learned are shown to give much better results then that of the hand crafted features thus the experimental results are encouraging.  The presentation of the sound net architecture is clear and most of the design choices are justified. Over all a well presented paper with incremental novelty but with good results.  The most crucial drawback of the paper is the absence of a reasonable baseline CNN which is trained on a large enough data-set without supervision. A simple soundnet based autoencoder as the unsupervised network should be trained on 2M training images as the baseline for acceptance of the paper.   My recommendation is a weak reject but I will be happy to change my rating if in the rebuttal authors can substantiate the utility of the teacher student setup of figure 1 over convolutional Auto encoder. 
The most crucial drawback of the paper is the absence of a reasonable baseline CNN which is trained on a large enough data-set.  The largest labelled sound dataset used for evaluation is ESC-10/50 with 2000 sound recordings which is too small for training a CNN from scratch. In table 1a) all methods are said to be hand crafted features except that of [24] which is supervised and trained on 2000 sound recordings?   I believe that the source if improvement over [24] could be  (i) the difference in architecture: this can be simply ruled out by training sound-net on ESC as a replacement of [24] and show that the sound-net is too deep to be trained directly on such small datasets. (ii) The knowledge transferred from state of the art visual recognition networks (Alex-net/VGG) as claimed by the paper. (iii) Simply because of the fact that we have now a large unlabelled dataset.  Given over 2M soundbites at our disposal, a very simple baseline is a traditional convolution autoencoder with tied weights similar to sound-net encoder. This encoder can be fine tuned on ECS and iHear separately for comparison. (A simple classifier on the autoencoder codes can also be used). This experiment is mandatory to verify weather the vision to audio knowledge transfer is useful. i.e how much improvement you get from merely using huge unlabelled data-set and quantify the contribution of visual training.  Another very important question is that flicker is said to mined with keywords. Although, these keywords does not correspond to the classification labels in the test experiment why can't we learn a siamese network with these surrogate labels and then use finetuning?     Other minor comments: - a brief description of these baselines is needed as the reader is bound to refer to line 47-48 even to know that out of so many baselines only [24] is a conv-net for which the architecture is not outlined in the paper). - As iHear seems to be more diverse and biggest labelled data-set, all the baseline (especially [24]) should be tested also on iHear.  - Why other unsupervised and supervised deep learning methods apart from [24] is not compared against?