We present a new unsupervised method for learning general-purpose sentence embeddings. Unlike existing methods which rely on local contexts, such as words inside the sentence or immediately neighboring sentences, our method selects, for each target sentence, influential sentences in the entire document based on a document structure. We identify a dependency structure of sentences using metadata or text styles. Furthermore, we propose a novel out-of-vocabulary word handling technique to model many domain-specific terms, which were mostly discarded by existing sentence embedding methods. We validate our model on several tasks showing 30% precision improvement in coreference resolution in a technical domain, and 7.5% accuracy increase in paraphrase detection compared to baselines.
This paper extends the idea of forming an unsupervised representation of sentences used in the SkipThought approach by using a broader set of evidence for forming the representation of a sentence. Rather than simply encoding the preceding sentence and then generating the next sentence, the model suggests that a whole bunch of related "sentences" could be encoded, including document title, section title, footnotes, hyperlinked sentences. This is a valid good idea and indeed improves results. The other main new and potentially useful idea is a new idea for handling OOVs in this context where they are represented by positional placeholder variables. This also seems helpful. The paper is able to show markedly better results on paraphrase detection that skipthought and some interesting and perhaps good results on domain-specific coreference resolution.
On the negative side, the model of the paper isn't very excitingly different. It's a fairly straightforward extension of the earlier SkipThought model to a situation where you have multiple generators of related text. There isn't a clear evaluation that shows the utility of the added OOV Handler, since the results with and without that handling aren't comparable. The OOV Handler is also related to positional encoding ideas that have been used in NMT but aren't reference. And the coreference experiment isn't that clearly described nor necessarily that meaningful. Finally, the finding of dependencies between sentences for the multiple generators is done in a rule-based fashion, which is okay and works, but not super neural and exciting.
Other comments:
 - p.3. Another related sentence you could possibly use is first sentence of paragraph related to all other sentences? (Works if people write paragraphs with a "topic sentence" at the beginning.
 - p.5. Notation seemed a bit non-standard. I thought most people use \sigma for a sigmoid (makes sense, right?), whereas you use it for a softmax and use calligraphic S for a sigmoid....
 - p.5. Section 5 suggests the standard way to do OOVs is to average all word vectors. That's one well-know way, but hardly the only way. A trained UNK encoding and use of things like character-level encoders is also quite common.
 - p.6. The basic idea of the OOV encoder seems a good one. In domain specific contexts, you want to be able to refer to and re-use words that appear in related sentences, since they are likely to appear again and you want to be able to generate them. A weakness of this section however is that it makes no reference to related work whatsoever. It seems like there's quite a bit of related work. The idea of using a positional encoding so that you can generate rare words by position has previously been used in NMT, e.g. Luong et al. (Google brain) (ACL 2015). More generally, a now quite common way to handle this problem is to use "pointing" or "copying", which appears in a number of papers. (e.g., Vinyals et al. 2015) and might also have been used here and might be expected to work too. 
 - p.7. Why such an old Wikipedia dump? Most people use a more recent one!
 - p.7. The paraphrase results seem good and prove the idea works. It's a shame they don't let you see the usefulness of the OOV model.
 - p.8. For various reasons, the coreference results seem less useful than they could have been, but they do show some value for the technique in the area of domain-specific coreference.