Neural Stored-program Memory
Neural networks powered with external memory simulate computer behaviors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures. The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus fully resemble the Universal Turing Machine or Von Neumann Architecture. A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks.
## Update I am changing my score to 7 (not adjusting the discretised score in the openreview interface though) Summary: The authors present the Neural Stored-program Memory (NSM), an architectural addition which is suitable for a large class of existing MANN models (NTM, DNC, LRUA etc). The model contains a second memory of key value pairs, which on each timestep can be read via cosine distance similarity on the keys, and for which the values contain all the weights necessary for the Interface Network (IN). The IN also receives the recurrent controller output, and produces an interface vector which is used to read or write to the regular memory. The definition is sufficiently general to allow the computation between an interface vector and the regular memory to be done according to NTM/DNC/etc schemes. NSM allows a MANN to effectively switch programs at every timestep, more closely matching classical computers and the Universal Turing Machine. The authors include a wide range of experiments, both showing faster learning on standard NTM tasks, and also introduce *Sequencing tasks* for which a single episode contains multiple instances of the standard NTM tasks in sequence. In principle this should allow the whole system to learn different programs for the subproblems, and dynamically switch between them as soon as the task changes. In principle a standard NTM could learn to do many of these tasks in a row, but the authors show that even when an NTM combined with NSM (denoted NUTM) has fewer trainable weights in total than a plain NTM, for some task sequence combinations the NUTM learns much faster. Experiments on continual learning, few shot learning and text based question answering back up the wide applicability of this technique, with many strong results including new SOTA on bAbI. Decision: Accept. Key reasons are that this is a relatively straightforward application of hypernetworks within a recurrent controller, which both has appealing justifications in the context of both Von Neumann and Turing models of computation. This simplicity is a positive, and the authors make a convincing argument that NSM can be applied to any MANN solving a sufficiently complex problem. A secondary reason is the extensive evaluation & hyperparam details, and while I do have some minor points on which I think the paper could be clarified (see below) I think this is overall a very nice paper. With the clarifications below addressed, I would give this paper a 7 but from the options I have available to choose from, 6 is the best fit for the current manuscript. Supporting arguments: The visualization of *program distribution* shows extremely clear phase changes as the underlying task changes - both within a single task (the reading phase vs writing phase for repeat copy) and across task boundaries (copy -> associative recall). Combined with the learning curves / results in the various tables, it is to me clear that the model is performing as designed. Despite the name of the *Meta Network*, and having a vague flavour of metalearning, the model does not require any elaborate *backprop through the inner training loop* of MAML et al, which is a benefit in my opinion. Additional feedback: Some of the experiments could be slightly more convincing - particularly Figure 5 which is lacking error bars. In my experience these architectures can have relatively high variance in performance, compared to other supervised domains, as evidenced by the spicy learning curves even in Figure 2 a). Error bars across multiple runs for figure 5 would be good, particularly for the points where the lines are close (eg after training on PS, the performance for C and RC is close for both models). The formatting of some figures and graphs could be improved: * Figure 1 - the graph could use some more text labels, rather than mathematical notation which needs to be referred to below. The colours are also slightly confusing - the program memory has slightly different shades of orange for the keys, slightly different shades of green for the values, whereas the regular memory has a slightly wider variety of colours. I was not sure at first whether this should be interpreted as indicating something important. Additionally, the value read from the NSM is reshaped to a 2x4 shape with various shades of blue, which then becomes the weights for a pink network? I think the colour adds little and may confuse people. There are some other issues, such as the value which should really come from the main memory as a result of the interface vector. With the supporting text, understanding the system is not hard, but I feel another pass over the diagram would benefit the camera ready version - consider ditching colour entirely, unless there is going to be some consistent meaning to things being the same colour vs not. Text labels with the various arrows (eg *vector used to lookup in NSM*, *used as network weights for Interface Network*) may improve clarity. should also be labelled on the diagram. * The y axis scale in figure 5 is very confusing - it took me several looks before I noticed that it goes from 50% to 100%, due to each number having digits in different vertical positions and different font sizes * Figure 2 y axis scale is a bit too small to easily read. * Figure 3: both read and write locations are shown on a single plot, but the green line that separates them is lamost unreadable on a printout. The task dependent, presumably manually chosen, approach to picking where this visualisation toggle should be made is a bit arbitrary - I would prefer to see read and write locations as separate subplots, as in the appendix. I found the exact details of number of programs versus number of heads (and the type of those heads) a bit confusing. In www.github.com/deepmind/dnc the code has *number of read heads* and *number of write heads* being two independently set integer parameters. This paper refers to * control heads*, but I am not exactly clear on how these are divded between read and write duties. Algorithm 1 references that write heads return zero on line 8 but not other mention of the two mutually exclusive types is made. The text towards the end of section 4.1 refers to the *no-read* strategy being assigned mostly to one program - this makes it sound like each program can (softly?) interpolate between reading or writing (or both?). The start of appendix B shows program distribution for read and write heads separately, but this then begs the question of what is happening in the examples for which only one program distribution is shown (eg Figure 3) - clearly we need to read and write for all of these tasks, so is one head with two programs doing both simultaneously? In the interests of reproducability, clarification here is essential. Separately, the decision of having a different program memory per control head is interesting - it*s not obvious to me why this would be necessary, surely one program memory would be sufficient as long as thre is a different (alg 1 line 5) network to choose a different program for the head? It would be good to see a line added to the paper justifying this choice. It seems like not all training hyperparams are specified in the appendix - eg the settings specified in Table 9 only apply to the few shot learning task, additional hyperparams are specified for bAbI, but I cannot see the training hyperparams for the experiments in sections 4.1 - 4.4. Minor correction: *As Turing Machine is finite state automata* -> *As Turing Machines are finite state automata*
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Neural Stored-program Memory
Neural networks powered with external memory simulate computer behaviors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures. The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus fully resemble the Universal Turing Machine or Von Neumann Architecture. A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



## Update I am changing my score to 7 (not adjusting the discretised score in the openreview interface though) Summary: The authors present the Neural Stored-program Memory (NSM), an architectural addition which is suitable for a large class of existing MANN models (NTM, DNC, LRUA etc). The model contains a second memory of key value pairs, which on each timestep can be read via cosine distance similarity on the keys, and for which the values contain all the weights necessary for the Interface Network (IN). The IN also receives the recurrent controller output, and produces an interface vector which is used to read or write to the regular memory. The definition is sufficiently general to allow the computation between an interface vector and the regular memory to be done according to NTM/DNC/etc schemes. NSM allows a MANN to effectively switch programs at every timestep, more closely matching classical computers and the Universal Turing Machine. The authors include a wide range of experiments, both showing faster learning on standard NTM tasks, and also introduce *Sequencing tasks* for which a single episode contains multiple instances of the standard NTM tasks in sequence. In principle this should allow the whole system to learn different programs for the subproblems, and dynamically switch between them as soon as the task changes. In principle a standard NTM could learn to do many of these tasks in a row, but the authors show that even when an NTM combined with NSM (denoted NUTM) has fewer trainable weights in total than a plain NTM, for some task sequence combinations the NUTM learns much faster. Experiments on continual learning, few shot learning and text based question answering back up the wide applicability of this technique, with many strong results including new SOTA on bAbI. Decision: Accept. Key reasons are that this is a relatively straightforward application of hypernetworks within a recurrent controller, which both has appealing justifications in the context of both Von Neumann and Turing models of computation. This simplicity is a positive, and the authors make a convincing argument that NSM can be applied to any MANN solving a sufficiently complex problem. A secondary reason is the extensive evaluation & hyperparam details, and while I do have some minor points on which I think the paper could be clarified (see below) I think this is overall a very nice paper. With the clarifications below addressed, I would give this paper a 7 but from the options I have available to choose from, 6 is the best fit for the current manuscript. Supporting arguments: The visualization of *program distribution* shows extremely clear phase changes as the underlying task changes - both within a single task (the reading phase vs writing phase for repeat copy) and across task boundaries (copy -> associative recall). Combined with the learning curves / results in the various tables, it is to me clear that the model is performing as designed. Despite the name of the *Meta Network*, and having a vague flavour of metalearning, the model does not require any elaborate *backprop through the inner training loop* of MAML et al, which is a benefit in my opinion. Additional feedback: Some of the experiments could be slightly more convincing - particularly Figure 5 which is lacking error bars. In my experience these architectures can have relatively high variance in performance, compared to other supervised domains, as evidenced by the spicy learning curves even in Figure 2 a). Error bars across multiple runs for figure 5 would be good, particularly for the points where the lines are close (eg after training on PS, the performance for C and RC is close for both models). The formatting of some figures and graphs could be improved: * Figure 1 - the graph could use some more text labels, rather than mathematical notation which needs to be referred to below. The colours are also slightly confusing - the program memory has slightly different shades of orange for the keys, slightly different shades of green for the values, whereas the regular memory has a slightly wider variety of colours. I was not sure at first whether this should be interpreted as indicating something important. Additionally, the value read from the NSM is reshaped to a 2x4 shape with various shades of blue, which then becomes the weights for a pink network? I think the colour adds little and may confuse people. There are some other issues, such as the value which should really come from the main memory as a result of the interface vector. With the supporting text, understanding the system is not hard, but I feel another pass over the diagram would benefit the camera ready version - consider ditching colour entirely, unless there is going to be some consistent meaning to things being the same colour vs not. Text labels with the various arrows (eg *vector used to lookup in NSM*, *used as network weights for Interface Network*) may improve clarity. should also be labelled on the diagram. * The y axis scale in figure 5 is very confusing - it took me several looks before I noticed that it goes from 50% to 100%, due to each number having digits in different vertical positions and different font sizes * Figure 2 y axis scale is a bit too small to easily read. * Figure 3: both read and write locations are shown on a single plot, but the green line that separates them is lamost unreadable on a printout. The task dependent, presumably manually chosen, approach to picking where this visualisation toggle should be made is a bit arbitrary - I would prefer to see read and write locations as separate subplots, as in the appendix. I found the exact details of number of programs versus number of heads (and the type of those heads) a bit confusing. In www.github.com/deepmind/dnc the code has *number of read heads* and *number of write heads* being two independently set integer parameters. This paper refers to * control heads*, but I am not exactly clear on how these are divded between read and write duties. Algorithm 1 references that write heads return zero on line 8 but not other mention of the two mutually exclusive types is made. The text towards the end of section 4.1 refers to the *no-read* strategy being assigned mostly to one program - this makes it sound like each program can (softly?) interpolate between reading or writing (or both?). The start of appendix B shows program distribution for read and write heads separately, but this then begs the question of what is happening in the examples for which only one program distribution is shown (eg Figure 3) - clearly we need to read and write for all of these tasks, so is one head with two programs doing both simultaneously? In the interests of reproducability, clarification here is essential. Separately, the decision of having a different program memory per control head is interesting - it*s not obvious to me why this would be necessary, surely one program memory would be sufficient as long as thre is a different (alg 1 line 5) network to choose a different program for the head? It would be good to see a line added to the paper justifying this choice. It seems like not all training hyperparams are specified in the appendix - eg the settings specified in Table 9 only apply to the few shot learning task, additional hyperparams are specified for bAbI, but I cannot see the training hyperparams for the experiments in sections 4.1 - 4.4. Minor correction: *As Turing Machine is finite state automata* -> *As Turing Machines are finite state automata*
assistant
## Update I am changing my score to 7 (not adjusting the discretised score in the openreview interface though) Summary: The authors present the Neural Stored-program Memory (NSM), an architectural addition which is suitable for a large class of existing MANN models (NTM, DNC, LRUA etc). The model contains a second memory of key value pairs, which on each timestep can be read via cosine distance similarity on the keys, and for which the values contain all the weights necessary for the Interface Network (IN). The IN also receives the recurrent controller output, and produces an interface vector which is used to read or write to the regular memory. The definition is sufficiently general to allow the computation between an interface vector and the regular memory to be done according to NTM/DNC/etc schemes. NSM allows a MANN to effectively switch programs at every timestep, more closely matching classical computers and the Universal Turing Machine. The authors include a wide range of experiments, both showing faster learning on standard NTM tasks, and also introduce *Sequencing tasks* for which a single episode contains multiple instances of the standard NTM tasks in sequence. In principle this should allow the whole system to learn different programs for the subproblems, and dynamically switch between them as soon as the task changes. In principle a standard NTM could learn to do many of these tasks in a row, but the authors show that even when an NTM combined with NSM (denoted NUTM) has fewer trainable weights in total than a plain NTM, for some task sequence combinations the NUTM learns much faster. Experiments on continual learning, few shot learning and text based question answering back up the wide applicability of this technique, with many strong results including new SOTA on bAbI. Decision: Accept. Key reasons are that this is a relatively straightforward application of hypernetworks within a recurrent controller, which both has appealing justifications in the context of both Von Neumann and Turing models of computation. This simplicity is a positive, and the authors make a convincing argument that NSM can be applied to any MANN solving a sufficiently complex problem. A secondary reason is the extensive evaluation & hyperparam details, and while I do have some minor points on which I think the paper could be clarified (see below) I think this is overall a very nice paper. With the clarifications below addressed, I would give this paper a 7 but from the options I have available to choose from, 6 is the best fit for the current manuscript. Supporting arguments: The visualization of *program distribution* shows extremely clear phase changes as the underlying task changes - both within a single task (the reading phase vs writing phase for repeat copy) and across task boundaries (copy -> associative recall). Combined with the learning curves / results in the various tables, it is to me clear that the model is performing as designed. Despite the name of the *Meta Network*, and having a vague flavour of metalearning, the model does not require any elaborate *backprop through the inner training loop* of MAML et al, which is a benefit in my opinion. Additional feedback: Some of the experiments could be slightly more convincing - particularly Figure 5 which is lacking error bars. In my experience these architectures can have relatively high variance in performance, compared to other supervised domains, as evidenced by the spicy learning curves even in Figure 2 a). Error bars across multiple runs for figure 5 would be good, particularly for the points where the lines are close (eg after training on PS, the performance for C and RC is close for both models). The formatting of some figures and graphs could be improved: * Figure 1 - the graph could use some more text labels, rather than mathematical notation which needs to be referred to below. The colours are also slightly confusing - the program memory has slightly different shades of orange for the keys, slightly different shades of green for the values, whereas the regular memory has a slightly wider variety of colours. I was not sure at first whether this should be interpreted as indicating something important. Additionally, the value read from the NSM is reshaped to a 2x4 shape with various shades of blue, which then becomes the weights for a pink network? I think the colour adds little and may confuse people. There are some other issues, such as the value which should really come from the main memory as a result of the interface vector. With the supporting text, understanding the system is not hard, but I feel another pass over the diagram would benefit the camera ready version - consider ditching colour entirely, unless there is going to be some consistent meaning to things being the same colour vs not. Text labels with the various arrows (eg *vector used to lookup in NSM*, *used as network weights for Interface Network*) may improve clarity. should also be labelled on the diagram. * The y axis scale in figure 5 is very confusing - it took me several looks before I noticed that it goes from 50% to 100%, due to each number having digits in different vertical positions and different font sizes * Figure 2 y axis scale is a bit too small to easily read. * Figure 3: both read and write locations are shown on a single plot, but the green line that separates them is lamost unreadable on a printout. The task dependent, presumably manually chosen, approach to picking where this visualisation toggle should be made is a bit arbitrary - I would prefer to see read and write locations as separate subplots, as in the appendix. I found the exact details of number of programs versus number of heads (and the type of those heads) a bit confusing. In www.github.com/deepmind/dnc the code has *number of read heads* and *number of write heads* being two independently set integer parameters. This paper refers to * control heads*, but I am not exactly clear on how these are divded between read and write duties. Algorithm 1 references that write heads return zero on line 8 but not other mention of the two mutually exclusive types is made. The text towards the end of section 4.1 refers to the *no-read* strategy being assigned mostly to one program - this makes it sound like each program can (softly?) interpolate between reading or writing (or both?). The start of appendix B shows program distribution for read and write heads separately, but this then begs the question of what is happening in the examples for which only one program distribution is shown (eg Figure 3) - clearly we need to read and write for all of these tasks, so is one head with two programs doing both simultaneously? In the interests of reproducability, clarification here is essential. Separately, the decision of having a different program memory per control head is interesting - it*s not obvious to me why this would be necessary, surely one program memory would be sufficient as long as thre is a different (alg 1 line 5) network to choose a different program for the head? It would be good to see a line added to the paper justifying this choice. It seems like not all training hyperparams are specified in the appendix - eg the settings specified in Table 9 only apply to the few shot learning task, additional hyperparams are specified for bAbI, but I cannot see the training hyperparams for the experiments in sections 4.1 - 4.4. Minor correction: *As Turing Machine is finite state automata* -> *As Turing Machines are finite state automata*