Expressing an Image Stream with a Sequence of Natural Sentences
We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their special moments, it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. To this end, we design a multimodal architecture called coherent recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional recurrent neural networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We demonstrate that our approach outperforms other state-of-the-art candidate methods, using both quantitative measures (e.g. BLEU and top-K recall) and user studies via Amazon Mechanical Turk.
The paper attacks the problem of describing a sequence of images from blog-posts with a sequence of consistent sentences. For this the paper proposes to first retrieve the K=5 most similar images and associated sentences from the training set for each query image. The main contribution of the paper lies in defining a way to select the most relevant sentences for the query image sequence, providing a coherent description. For this sentences are first embedded in a vector and then the sequence of sentences is modeled with a bidirectional LSTM. The output of the bi-directional LSTM is first fed through a relu and fully connected layer and then scored with a compatibility score between image and sentence. Additionally a local coherence model [1] is included to enforce the compatibility between sentences.   Strength / positive aspects: - The paper proposes a novel and effective architecture to retrieve coherent sentences for an image sequence. - The paper provides an extensive quantitative, qualitative, and human evaluation, showing the superiority of their approach against several baselines, not using coherence. They also provide an ablation experiment, removing the coherence model [1]. - The authors promise to release source code and dataset. Weaknesses / Questions / Unclarities: 1. Line 94: The paper claims that there is "no mechanism for the coherence between sentences" in [5]. Although not the contribution of [5], [5] predicts an intermediate semantic representation of videos, which is coherent across sentences by modeling the topic of the multi-sentence description. 2. Correctness/clarity: Figure 2b does not seem to correspond to the description 3.3/Equation (2). While Figure 2b implies that the fully connected layer are connected to all sentences, this it not the case in Eq2, which implies that the parameters are shared across sentences, but only connected to the vector representing a single sentence. 3. A better metric to automatically evaluate the generated sentences is Meteor (http://www.cs.cmu.edu/~alavie/METEOR/) instead of BLEU, especially if there is only a single reference sentence. 4. Why two linear functions in Eq2 (W_{f2}, W_{f1}) are applied behind each other? Given that two linear functions are again a linear function the benefit is unclear. An ablation study showing the benefit of these functions would be interesting. 5. Why the same parameters of the fully connected layers are used for the BRNN output (o_t) and the local coherence model q (Equation 2)? 6. Is the paragraph vector [16] fine-tuned or kept fixed? === post rebuttal === After reading the rebuttal I recommend the paper for acceptance. The authors successfully addressed issues with the formulation, evaluation, and related work.Please make the promised changes to the final and also clarify the following point in the final. 6. Is the paragraph vector [16] fine-tuned or kept fixed?