Unsupervised Learning of Spoken Language with Visual Context
Humans learn to speak before they can read or write, so why can't computers do the same? In this paper, we present a deep neural network model capable of rudimentary spoken language acquisition using untranscribed audio training data, whose only supervision comes in the form of contextually relevant visual images. We describe the collection of our data comprised of over 120,000 spoken audio captions for the Places image dataset and evaluate our model on an image search and annotation task. We also provide some visualizations which suggest that our model is learning to recognize meaningful words within the caption spectrograms.
The paper presents a multi-modal neural network that learns similarities between images (or VGG objects therein) and audio captions that Turkers produced for these images. The idea is that a network would learn what "car" looks like in the audio, if the image shows a "car". While the problem of semantic "grounding" has been around for a long time, the intersection of "language and vision" and deep learning presents a new angle to this problem, which the present paper (and other works) are trying to exploit.
This is interesting work that is pointing into the right direction, but a few aspects of this paper are a bit problematic:  1) It would have been useful (or interesting) to use a corpus that has existing text captions, and either have users re-speak the text captions, or collect additional captions. The data collections seems generally well thought-out, but why was the Places205 data set used? Because of the focus on "objects"? Prompted speech (such as collected here) is not "spontaneous", otherwise the WSJ recognizer would not have given 20 % WER (this aspect is irrelevant for the purpose of this paper, though, I think). Typically, multiple captions are being generated for a single image. Has this been done here as well? Or is there only a single caption for each image? Why?   2) The presentation of the speech/ audio work could be a bit clearer: put all WER results (including DEL/INS/SUB) in a table, present an analysis of the most frequent words for each error type. How much data (in time and/ or words) is 100 captions? < 15 minutes? Is it possible to diagnose more clearly what the difference between the Google and Kaldi recognizers are? How big is the silence portion of the data? How was the vocabulary size (27,891) determined without manual transcription? Was there some post-segmentation of any kind to remove excess silence? (If there is any?) Which fraction of the data was clipped to 10 seconds? Which fraction was padded? Would it be possible to "sub-sample" frames, by iteratively dropping frames that are very similar to their neighbor, until 1000 frames have been reached? Is it necessary to use a CNN, rather than a simple DNN? Does it help? Has VTLN/ feature adaptation been applied? You could get those from the Kaldi recipe, I presume. (Note that the last few items are suggestions and certainly not required in the first publication in the topic)  3) The model seems a bit simplistic, although average pooling and a DNN/CNN architecture is certainly a viable first approach. I like the analysis of drawing parallels in the evaluation to see what the level of performance is with respect to other corpora and/ or tasks, but there are just too many differences. It might have been better to try more variants of the present model to gain insight into what factors influence the performance, and which ones don't (e.g. CNN vs DNN, silence segmentation approach, features, etc.). While the overall story and presentation of the work is good, the third sentence in Figure 2 contains BASEBALL twice. The first time, it is relevant (according to the similarity profile), the second time, it is not. This effect should be analyzed. Is this a feature of the model? Or an artifact that would disappear with better training etc?   It is good that the authors plan to release their data (maybe even with a baseline system).  The references seem good (the authors should check for other relevant work before publication, though).  Finally, the paper has many small typos and grammar errors - please fix.