Learning to Transduce with Unbounded Memory
Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.
This paper presents neural stack, queue, and neural deque data structures with the aim of drawing the 'attention' of the network to the relevant memory/information in the past. More specifically, the state of the data structure (its memory content and the strength signal) is updated by control signals from a 'controller'. The controller is the core part of the model which 'reads' the relevant information from the past from the neural data structure, and combines it with its previous state and the input to generate the output and the control signal to change the neural data structure content for the next step. The paper presents experimental results on synthetic data showing that the neural data structures provide suitable structured memory to draw the 'attention' of the model to the relevant part of the history. The experiments are performed across a range of transduction tasks, and compared with LSTM as the baseline.I like the paper and I believe it present interesting neural data structures to implement tractable structured memory networks. However, there are some issues there: - The presentation of the model is not good, i.e. it is very difficult to understand. Authors need to improve the presentation to make it understandable for more people.- Authors claim the neural Turing machines are intractable, hence justifying the need for more restricted but tractable models of computation. They need to show the intractability of neural Turing machines and the tractability of neural data structures on some simple experiments to back up their claim.- Authors need to compare in more details with the Facebook work, both in the 'Related Work' section AND in the experiments section. I would very much like to see how these works are compared empirically. - Why the number of baseline LSTM layers is different across tasks and how it has been specified for each task? - Finally, authors need to show the usefulness of the neural data structures on real life datasets, even small. For example, they could have used BTEC for machine translation, if they cannot run the models on large bilingual corpora.I would very much appreciate authors' comments on the above issues.