EVIDENCE AGGREGATION FOR ANSWER RE-RANKING IN OPEN-DOMAIN QUESTION ANSWERING
A popular recent approach to answering open-domain questions is to first search for question-related passages and then apply reading comprehension models to extract answers. Existing methods usually extract answers from single passages independently. But some questions require a combination of evidence from across different sources to answer correctly. In this paper, we propose two models which make use of multiple passages to generate their answers. Both use an answerreranking approach which reorders the answer candidates generated by an existing state-of-the-art QA model. We propose two methods, namely, strengthbased re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer.
Traditional open-domain QA systems typically have two steps: passage retrieval and aggregating answers extracted from the retrieved passages.  This paper essentially follows the same paradigm, but leverages the state-of-the-art reading comprehension models for answer extraction, and develops the neural network models for the aggregating component.  Although the idea seems incremental, the experimental results do seem solid.  The paper is generally easy to follow, but in several places the presentation can be further improved.
Detailed comments/questions:
  1. In Sec. 2.2, the justification for adding H^{aq} and \bar{H}^{aq} is to downweigh the impact of stop word matching.  I feel this is a somewhat indirect and less effective design, if avoiding stop words is really the reason.  A standard preprocessing step may be better.
  2. In Sec. 2.3, it seems that the final score is just the sum of three individual normalized scores. It's not truly a "weighted" combination, where the weights are typically assumed to be tuned.
  3. Figure 3: Connecting the dots in the two subfigures on the right does not make sense.  Bar charts should be used instead.
  4. The end of Sec. 4.2: I feel it's a bad example, as the passage does not really support the answer. The fact that "Sesame Street" got picked is probably just because it's more famous.
  5. It'd be interesting to see how traditional IR answer aggregation methods perform, such as simple classifiers or heuristics by word matching (or weighted by TFIDF) and counting.  This will demonstrates the true advantages of leveraging modern NN models.
Pros:
  1. Updating a traditional open-domain QA approach with neural models
  2. Experiments demonstrate solid positive results
Cons:
  1. The idea seems incremental
  2. Presentation could be improved