Transformers have obtained significant success modeling natural language as a sequence of text tokens. However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents. This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way. Its new attention mechanism naturally "hops" across the connected text sequences in addition to attending over tokens within each sequence. Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence. This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.
This paper introduces the Transformer XH model architecture, a transformer architecture that scales better to long sequences / multi-paragraph text compared to the standard transformer architecture. As I understand, Transformer XH is different from the standard transformer architecture in two main ways: 1) adding attention across paragraphs (or sub-sequences) via the CLS token and 2) defining the structure of that attention based on the entities in the paragraphs (or sub-sequences). The paper tackles an important problem (learning from long sequences) and achieves good empirical results on HotpotQA.
Modification #1 has already been explored by previous works which should be discussed in the paper. "Generating Wikipedia by Summarizing Long Sequences" by Liu, Saleh, et al. propose the T-DMCA model, with a similar motivation: enabling the transformer to work on/scale to longer sequences. T-DMCA and Transformer XH have some difference (T-DMCA seems to have more capacity while Transformer XH is simpler); I think it is necessary to compare against Transformer XH against T-DMCA on HotpotQA to know whether a new architecture is really necessary for HotpotQA. "Generating Long Sequences with Sparse Transformers" from Child et al. also proposes another general transformer architecture that can handle long sequences, and it would be ideal to compare against this architecture as well. Sparse Transformers reduce the time complexity of attention to reduce O(n√ n), which seems similar to the reduction that Transformer XH gets.
For modification #2 (defining the attention structure beforehand using e.g. entity linking), it does not seem too difficult to learn the attention structure directly instead, as confirmed by the ablation in Table 3 which uses a fully connected graph structure. A model that learned the attention pattern or used a fully connected graph would be more general (but more similar to T-DMCA and sparse transformers).
The empirical results are good. It's nice that a simple/straightforward architecture like Transformer XH works quite well (compared to some previous approaches which were not as elegant). However, I do not feel that prior work has explored the best transformer architectures for HotpotQA (such as Sparse Transformers or T-DMCA), and since this work is specifically proposing a new transformer architecture, I think it is important to compare directly against other transformer architectures. Other (previously) SOTA models like SR-MRS are quite simple (conceptually), so it's likely that such models will also be outperformed by transformer architectures that are better adapted to long sequences. In general, I think that the most relevant baseline already present is SR-MRS rather than CogQA; the fact that such a simple approach like SR-MRS works well is an important fact about HotpotQA to take into account (even if SR-MRS is concurrent). In other words, SR-MRS shows that previous models like CogQA are likely weaker baselines.
Because of the missing baselines and limited novelty compared to prior work, I overall lean towards rejecting the paper (despite the good empirical results).
I do have a few specific questions for the authors:
- Would you mind providing further details about the following sentence? "For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set Dir ∪ Del and keep top K ranked ones."
- Is only answer-level supervision used? Or is supporting-fact level supervision used to train any of the rankers or pipeline?