To solve the text-based question and answering task that requires relational reasoning, it is necessary to memorize a large amount of information and find out the question relevant information from the memory. Most approaches were based on external memory and four components proposed by Memory Network. The distinctive component among them was the way of finding the necessary information and it contributes to the performance. Recently, a simple but powerful neural network module for reasoning called Relation Network (RN) has been introduced. We analyzed RN from the view of Memory Network, and realized that its MLP component is able to reveal the complicate relation between question and object pair. Motivated from it, we introduce Relation Memory Network (RMN) which uses MLP to find out relevant information on Memory Network architecture. It shows new state-of-the-art results in jointly trained bAbI-10k story-based question answering tasks and bAbI dialog-based question answering tasks.
This paper introduces Related Memory Network (RMN), an improvement over Relationship Networks (RN). RMN avoids growing the relationship time complexity as suffered by RN (Santoro et. Al 2017). RMN reduces the complexity to linear time for the bAbi dataset. RN constructs pair-wise interactions between objects in RN to solve complex tasks such as transitive reasoning. RMN instead uses a multi-hop attention over objects followed by an MLP to learn relationships in linear time.
Comments for the author:
The paper addresses an important problem since understanding object interactions are crucial for reasoning. However, how widespread is this problem across other models or are you simply addressing a point problem for RN? For example, Entnet is able to reason as the input is fed in and the decoding costs are low. Likewise, other graph-based networks (which although may require strong supervision) are able to decode quite cheaply. 
The relationship network considers all pair-wise interactions that are replaced by a two-hop attention mechanism (and an MLP). It would not be fair to claim superiority over RN since you only evaluate on bABi while RN also demonstrated results on other tasks. For more complex tasks (even over just text), it is necessary to show that you outperform RN w/o considering all objects in a pairwise fashion. More specifically, RN uses an MLP over pair-wise interactions, does that allow it to model more complex interactions than just selecting two hops to generate attention weights. Showing results with multiple hops (1,2,..) would be useful here.
More details are needed about Figure 3. Is this on bAbi as well? How did you generate these stories with so many sentences? Another clarification is the bAbi performance over Entnet which claims to solve all tasks. Your results show 4 failed tasks, is this your reproduction of Entnet?
Finally, what are the savings from reducing this time complexity? Some wall clock time results or FLOPs of train/test time should be provided since you use multiple hops.
Overall, this paper feels like a small improvement over RN. Without experiments over other datasets and wall clock time results, it is hard to appreciate the significance of this improvement. One direction to strengthen this paper is to examine if RMN can do better than pair-wise interactions (and other baselines) for more complex reasoning tasks.