Processing sequential data with long term dependencies and learn complex transitions are two major challenges in many deep learning applications. In this paper, we introduce a novel architecture, the Auto-addressing and Recurrent Memory Integrating Network (ARMIN) to address these issues. The ARMIN explicitly stores previous hidden states and recurrently integrate useful past states into current time-step by an efficient memory addressing mechanism. Compared to existing memory networks, the ARMIN is more light-weight and inference-time efficient. Our network can be trained on small slices of long sequential data, and thus, can boost its training speed. Experiments on various tasks demonstrate the efficiency of the ARMIN architecture. Codes and models will be available.
This paper introduces a memory-augmented RNN (MARNN) which aims at being lightweight and   differentiable. In a nutshell, authors propose to augment a LSTM-type architecture with several memory cells. At each time-step, MARNN retrieves one memory cell, updates his state, and updates the memory cell content. To learn the retrieval operation that requires discrete addressing,  authors rely on the Gumbel-Softmax. Authors evaluate their approach on PennTreeBank character level modelling where they demonstrate competitive performances. They also report state-of-art performance on the Thumos dataset. The paper is overall clear and pleasant to read. 
Authors highlight that MARNN is more lightweight compared to existing memory networks. MARNN can indeed retrieves only memory cells at inference. However,  since MARNN uses a Gumbel-Softmax to train the discrete addressing scheme, it is it not clear if there is any advantage in term of memory and computation of MARNN relatively to other network during training? It would be nice to compare the computation time/memory usage of MARNN with other memory augmented network such as TARDIS, NTM or Memory Network during training and inference. 
Another claim is that MARNN can possibly boost training speed by reducing the lengths of TBTT.  But MARNN also haves a training time overhead as showed in Figure 2.  How does the overall training time/performances of MARNN with TBPTT of 50 compared to a LSTM with TBPTT of 100/150?
The writing can be sometime a bit imprecise. For instance authors say that MARNN “learns better representations that many hierarchical RNN structure”. I agree that MARNN outperforms in term of accuracy, however, it is not clear what the author are referring to by “better representation” of the MARNN hidden state? Performance gain of MARNN could also be due to the external memory which allows  to retain more information of the input? In addition, it would be nice to precise which type of hierarchical RNN structure MARNN does (or doesn’t) outperform. Another claim is that MARNN can “easily learn long-term dependencies”. While this is reasonable, I am unsure that the empirical evaluation support this.  It would be nice to show how the gradients backpropagated through time behave in practice to support this claim? 
Memory-augmented network are a very important research directions and the MARNN architecture is interesting. However, it is not entirely clear to me what is the main advantage of MARNN relatively to other memory networks network such as TARDIS, NTM or Memory Network for training and/or inference. Although authors do compare with TARDIS, further comparison with the other networks and in term of computation time and memory could help clarify those points.