Text embedding representing natural language documents in a semantic vector space can be used for document retrieval using nearest neighbor lookup. In order to study the feasibility of neural models specialized for retrieval in a semantically meaningful way, we suggest the use of the Stanford Question Answering Dataset (SQuAD) in an open-domain question answering context, where the first task is to find paragraphs useful for answering a given question. First, we compare the quality of various text-embedding methods on the performance of retrieval and give an extensive empirical comparison on the performance of various non-augmented base embedding with, and without IDF weighting. Our main results are that by training deep residual neural models specifically for retrieval purposes can yield significant gains when it is used to augment existing embeddings. We also establish that deeper models are superior to this task. The best base baseline embeddings augmented by our learned neural approach improves the top-1 recall of the system by 14% in terms of the question side, and by 8% in terms of the paragraph side.
This paper tries to study retrieval methods for multi-paragraph / multi-document reading comprehension.  The basic approach is to embed the question and the paragraph and train a system to put the correct paragraph close to the question.  I had a very hard time following the details of the proposed approach for this, however, and I still don't really understand what the authors are proposing.
This paper is not ready for publication.  The exposition is not at all clear and needs substantial rewriting.  Additionally, the evaluation done in the paper is not well-justified.  I do not know what "paragraph-side" means, but I assume that means you are trying to retrieve the question given the paragraph.  Why?  There were no standard baselines compared against, like a simple IR system (Lucene).  And I expected to see actual impact of the retrieved results on downstream QA performance of a system like Chen et al.'s, or Clark and Gardner 2018.  Even if you have a slightly better ranking of the retrieved paragraphs, it's not clear to me that this will improve performance, if the downstream method is properly calibrated to handle multiple paragraphs (see Clark and Gardner 2018).
A few writing suggestions for the authors, for next time:
This paper does not follow the typical flow of an academic paper.  It reads too much like a logbook of what you did, presented chronologically, instead of presenting the ideas in a coherent sequence.  Part of this is just simple wording fixes (e.g., avoid things like "it was time to compute ELMo representations" - this isn't a logbook).  Also, all of the shape comments and numerical details at the top of page 4 are out of place.  Describe your method first in general terms, then give experimental details (like corpus size, etc.) later.  I suggest reading the award-winning papers at various conferences to get a sense of how these papers are typically structured and phrased.
Section 2: A full page dedicated to the history of word embeddings is entirely unnecessary for this paper.  This is not a survey on word embeddings.  It's much more useful to the reader to give pointers to multiple connection points between your work and the rest of the literature.  You could have given a paragraph to the most relevant embedding techniques, a paragraph to the most relevant retrieval / multi-paragraph techniques (e.g., Clark and Gardner 2018, which is very relevant, along with Chen et al., TriviaQA, others), and a paragraph to distance metric learning.