Vector semantics, especially sentence vectors, have recently been used successfully in many areas of natural language processing. However, relatively little work has explored the internal structure and properties of spaces of sentence vectors. In this paper, we will explore the properties of sentence vectors by studying a particular real-world application: Automatic Summarization. In particular, we show that cosine similarity between sentence vectors and document vectors is strongly correlated with sentence importance and that vector semantics can identify and correct gaps between the sentences chosen so far and the document. In addition, we identify specific dimensions which are linked to effective summaries. To our knowledge, this is the first time specific dimensions of sentence embeddings have been connected to sentence properties. We also compare the features of different methods of sentence embeddings. Many of these insights have applications in uses of sentence embeddings far beyond summarization.
This paper examines a number of sentence and document embedding methods for automatic summarization. It pairs a number of recent sentence embedding algorithms (e.g., Paragraph Vectors and Skip-Thought Vectors) with several simple summarization decoding algorithms for sentence selection, and evaluates the resulting output summary on DUC 2004 using ROUGE, based on the general intuition that the selected summary should be similar to the original document in the vector space induced by the embedding algorithm. It further provides a number of analyses of the sentence representations as they relate to summarization, and other aspects of the summarization process including the decoding algorithm.
The paper was well written and easy to understand. I appreciate the effort to apply these representation techniques in an extrinsic task.
However, the signficance of the results may be limited, because the paper does not respond to a long line of work in summarization literature which have addressed many of the same points. In particular, I worry that the paper may in part be reinventing the wheel, in that many of the results are quite incremental with respect to previous observations in the field.
Greedy decoding and non-redundancy: many methods in summarization use greedy decoding algorithms. For example, SumBasic (Nenkova and Vanderwende, 2005), and HierSum (Haghighi and Vanderwende, 2009) are two such papers. This specific topic has been thoroughly expanded on by the work on greedy decoding for submodular objective functions in summarization (Lin and Bilmes, 2011), as well as many papers which focus on how to optimize for both informativeness and non-redundancy (Kulesza and Taskar, 2012). 
	
The idea that the summary should be similar to the entire document is known as centrality. Some papers that exploit or examine that property include (Nenkova and Vanderwende, 2005; Louis and Nenkova, 2009; Cheung and Penn, 2013)
 
Another possible reading of the paper is that its novelty lies in the evaluation of sentence embedding models, specifically. However, these methods were not designed for summarization, and I don't see why they should necessarily work well for this task out of the box with simple decoding algorithms without finetuning. Also, the ROUGE results are so far from the SotA that I'm not sure what the value of analyzing this suite of techniques is.
  
In summary, I understand that this paper does not attempt to produce a state-of-the-art summarization system, but I find it hard to understand how it contributes to our understanding of future progress in the summmarization field. If the goal is to use summarization as an extrinsic evaluation of sentence embedding models, there needs to be better justification of this is a good idea when there are so many other issues in content selection that are not due to sentence embedding quality, but which affect summarization results.
References:
Nenkova and Vanderwende, 2005. The impact of frequency on summarization. Tech report.
Haghighi and Vanderwende, 2009. Exploring content models for multi-document summarization. NAACL-HLT 2009.
Lin and Bilmes, 2011. A class of submodular functions for document summarization. ACL-HLT 2011.
Kulesza and Taskar, 2012. Learning Determinantal Point Processes.
Louis and Nenkova, 2009. Automatically evaluating content selection in summarization without human models. EMNLP 2009.
Cheung and Penn, 2013. Towards Robust Abstractive Multi-Document Summarization: A Caseframe Analysis of Centrality and Domain. ACL 2013.
Other notes:
The acknowledgements seem to break double-blind reviewing.