End-To-End Memory Networks
We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network [23] but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch [2] to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering [22] and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.
Summary:This paper presents an end-to-end version of memory networks (Weston et al., 2015) such that the model doesn't train on the intermediate 'supporting facts' strong supervision of which input sentences are the best memory accesses, making it much more realistic. They also have multiple hops (computational steps) per output symbol. The tasks are Q&A and language modeling, and achieves strong results.Quality: The paper is a useful extension of memNN because it removes the strong, unrealistic supervision requirement and still performs pretty competitively. The architecture is defined pretty cleanly and simply. The related work section is quite well-written, detailing the various similarities and differences with multiple streams of related work. The discussion about the model's connection to RNNs is also useful. My main concern with the paper is that it presents various model decisions and choices without motivating them, without empirically ablating them, and also without apparently using a dev set for most of these decisions (and hence tuning and presenting too many results on the test set directly?). Some examples are: -- Eqn3 -- the various wight tying choices, and whether these generalize to other tasks (Q&A or otherwise) -- temporal encoding design choices -- the various training decisions in Sec 4.2 and Sec 4.4 -- Table 1, that does finally have ablations for some remaining, more important choices, is done directly on the test set, at the risk of reporting too many numbers on the test set and essentially choosing the best among those as their final number to compare to baselines. Ideally, all these decisions should be made on a dev/validation set and then the final setting choice should be run on test.Other issues or suggestions: -- Encoding with an RNN or LSTM-RNN would be good to compare to the current approach of BoW or temporal encoding. -- Also, for the MemNN-WSH baseline, why not have a better classifier to predict which sentences to use, instead of using >1 word match? -- the perplexity differences in Table2 from the LSTM baseline do not look large so authors should report statistical significance results. The gap might be even smaller or negative if the current model did not so many extra parameters to tune over, as compared to LSTMs. -- perplexity in language modeling is not very convincing as a metric, so maybe useful to also apply this model (and show its generalizability) to text categorization tasks like tagging, NER, sentiment, etc.Clarity: The paper is well-written and well-organizedOriginality and significance: The paper is original and has a useful impact in terms of moving away from the strong supervision assumption of memNNs, and using multiple hops per output; the related work is well-written to clarify the originality. However, there are issues with model tuning and decisions, comparison, and need of better tasks. ########## Post-rebuttal: Thanks for answering some of my questions. I have increased my score to a 7 but please include all the dev/test details, model choice ablations, and statistical significance tests for perplexity. Also try to add some non-perplexity metrics.