Learning Word-Like Units from Joint Audio-Visual Analylsis
Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the words ``lighthouse** within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.
This work proposes a joint classification of images and audio captions for the task of word like discovery of acoustic units that correlate to semantically visual objects. The general this is a very interesting direction of research as it allows for a richer representation of data: regularizing visual signal with audio and visa versa. This allows for training of visual models from video, etc. A major concern is the amount of novelty between this work and the author*s previous publication at NIPs 2016. The authors claim a more sophisticated architecture and indeed show an improvement in recall. However, the improvements are marginal, and the added complexity to the architecture is a bit ad hoc. Clustering and grouping in section 4, is hacky. Instead of gridding the image, the authors could actually use an object detector (SSD, Yolo, FasterRCNN, etc.) to estimate accurate object proposals; rather than using k-means, a spectral clustering approach would alleviate the gaussian assumption of the distributions. In assigning visual hypotheses with acoustic segments, some form of bi-partite matching should be used. Overall, I really like this direction of research, and encourage the authors to continue developing algorithms that can train from such multimodal datasets. However, the work isn*t quite novel enough from NIPs 2016.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Word-Like Units from Joint Audio-Visual Analylsis
Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the words ``lighthouse** within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work proposes a joint classification of images and audio captions for the task of word like discovery of acoustic units that correlate to semantically visual objects. The general this is a very interesting direction of research as it allows for a richer representation of data: regularizing visual signal with audio and visa versa. This allows for training of visual models from video, etc. A major concern is the amount of novelty between this work and the author*s previous publication at NIPs 2016. The authors claim a more sophisticated architecture and indeed show an improvement in recall. However, the improvements are marginal, and the added complexity to the architecture is a bit ad hoc. Clustering and grouping in section 4, is hacky. Instead of gridding the image, the authors could actually use an object detector (SSD, Yolo, FasterRCNN, etc.) to estimate accurate object proposals; rather than using k-means, a spectral clustering approach would alleviate the gaussian assumption of the distributions. In assigning visual hypotheses with acoustic segments, some form of bi-partite matching should be used. Overall, I really like this direction of research, and encourage the authors to continue developing algorithms that can train from such multimodal datasets. However, the work isn*t quite novel enough from NIPs 2016.
assistant
This work proposes a joint classification of images and audio captions for the task of word like discovery of acoustic units that correlate to semantically visual objects. The general this is a very interesting direction of research as it allows for a richer representation of data: regularizing visual signal with audio and visa versa. This allows for training of visual models from video, etc. A major concern is the amount of novelty between this work and the author*s previous publication at NIPs 2016. The authors claim a more sophisticated architecture and indeed show an improvement in recall. However, the improvements are marginal, and the added complexity to the architecture is a bit ad hoc. Clustering and grouping in section 4, is hacky. Instead of gridding the image, the authors could actually use an object detector (SSD, Yolo, FasterRCNN, etc.) to estimate accurate object proposals; rather than using k-means, a spectral clustering approach would alleviate the gaussian assumption of the distributions. In assigning visual hypotheses with acoustic segments, some form of bi-partite matching should be used. Overall, I really like this direction of research, and encourage the authors to continue developing algorithms that can train from such multimodal datasets. However, the work isn*t quite novel enough from NIPs 2016.