Extensions and Limitations of the Neural GPU
The Neural GPU is a recent model that can learn algorithms such as multi-digit binary addition and binary multiplication in a way that generalizes to inputs of arbitrary length. We show that there are two simple ways of improving the performance of the Neural GPU: by carefully designing a curriculum, and by increasing model size. The latter requires a memory efficient implementation, as a naive implementation of the Neural GPU is memory intensive. We find that these techniques increase the set of algorithmic problems that can be solved by the Neural GPU: we have been able to learn to perform all the arithmetic operations (and generalize to arbitrarily long numbers) when the arguments are given in the decimal representation (which, surprisingly, has not been possible before). We have also been able to train the Neural GPU to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of the operands, although these have succeeded only in their binary representation, and not with perfect accuracy. In addition, we gain insight into the Neural GPU by investigating its failure modes. We find that Neural GPUs that correctly generalize to arbitrarily long numbers still fail to compute the correct answer on highly-symmetric, atypical inputs: for example, a Neural GPU that achieves near-perfect generalization on decimal multiplication of up to 100-digit long numbers can fail on while succeeding at . These failure modes are reminiscent of adversarial examples.
Overall the paper has the feel of a status update by some of the best researchers in the field. The paper is very clear, the observations are interesting, but the remarks are scattered and don*t add up to a quantum of progress in the study of what can be done with the Neural GPU model. Minor remark on the use of the term RNN in Table 1: I found Table 1 confusing because several of the columns are for models that are technically RNNs, and use of RNNs for e.g. translation and word2vec highlight that RNNs can be characterized in terms of the length of their input sequence, the length of their input, and the sizes (per step) of their input, output, and working memories. Basic model question: How are inputs presented (each character 1-hot?) and outputs retrieved when there are e.g. 512 “filters” in the model ? If inputs and outputs are 1-hot encoded, and treated with the same filters as intermediate layers, then the intermediate activation functions should be interpretable as digits, and we should be able to interpret the filters as implementing a reliable e.g. multiplication-with-carry algorithm. Looking at the intermediate values may shed some light on why the usually-working models fail on e.g. the pathological cases identified in Table 3. The preliminary experiment on input alignment is interesting in two ways: the seeds for effective use of an attentional mechanism are there, but also, it suggests that the model is not presently dealing with general expression evaluation the way a correct algorithm should. The remarks in the abstract about improving the memory efficiency of Neural GPU seem overblown -- the paragraph at the top of page 6 describes the improvements as using tf.while_loop instead of unrolling the graph, and using swap_memory to use host memory when GPU memory runs short. These both seem like good practice, but not a remarkable improvement to the efficiency of the model, in fact it would likely slow down training and inference when memory does in fact fit in the GPU. The point about trying many of random seeds to get convergence makes me wonder if the Neural GPU is worth its computational cost at all, when evaluated as means of learning algorithms that are already well understood (e.g. parsing and evaluating S-exprs). Consider spending all of the computational cycles that go into training one of these models (with the multiple seeds) on a traditional search through program space (e.g. sampling lisp programs or something). The notes on the curriculum strategies employed to get the presented results were interesting to read, as an indication of the lengths to which someone might have to go to train this sort of model, but it does leave this reviewer with the impression that despite the stated extensions of the Neural GPU model it remains unclear how useful it might be to practical problems.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Extensions and Limitations of the Neural GPU
The Neural GPU is a recent model that can learn algorithms such as multi-digit binary addition and binary multiplication in a way that generalizes to inputs of arbitrary length. We show that there are two simple ways of improving the performance of the Neural GPU: by carefully designing a curriculum, and by increasing model size. The latter requires a memory efficient implementation, as a naive implementation of the Neural GPU is memory intensive. We find that these techniques increase the set of algorithmic problems that can be solved by the Neural GPU: we have been able to learn to perform all the arithmetic operations (and generalize to arbitrarily long numbers) when the arguments are given in the decimal representation (which, surprisingly, has not been possible before). We have also been able to train the Neural GPU to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of the operands, although these have succeeded only in their binary representation, and not with perfect accuracy. In addition, we gain insight into the Neural GPU by investigating its failure modes. We find that Neural GPUs that correctly generalize to arbitrarily long numbers still fail to compute the correct answer on highly-symmetric, atypical inputs: for example, a Neural GPU that achieves near-perfect generalization on decimal multiplication of up to 100-digit long numbers can fail on while succeeding at . These failure modes are reminiscent of adversarial examples.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Overall the paper has the feel of a status update by some of the best researchers in the field. The paper is very clear, the observations are interesting, but the remarks are scattered and don*t add up to a quantum of progress in the study of what can be done with the Neural GPU model. Minor remark on the use of the term RNN in Table 1: I found Table 1 confusing because several of the columns are for models that are technically RNNs, and use of RNNs for e.g. translation and word2vec highlight that RNNs can be characterized in terms of the length of their input sequence, the length of their input, and the sizes (per step) of their input, output, and working memories. Basic model question: How are inputs presented (each character 1-hot?) and outputs retrieved when there are e.g. 512 “filters” in the model ? If inputs and outputs are 1-hot encoded, and treated with the same filters as intermediate layers, then the intermediate activation functions should be interpretable as digits, and we should be able to interpret the filters as implementing a reliable e.g. multiplication-with-carry algorithm. Looking at the intermediate values may shed some light on why the usually-working models fail on e.g. the pathological cases identified in Table 3. The preliminary experiment on input alignment is interesting in two ways: the seeds for effective use of an attentional mechanism are there, but also, it suggests that the model is not presently dealing with general expression evaluation the way a correct algorithm should. The remarks in the abstract about improving the memory efficiency of Neural GPU seem overblown -- the paragraph at the top of page 6 describes the improvements as using tf.while_loop instead of unrolling the graph, and using swap_memory to use host memory when GPU memory runs short. These both seem like good practice, but not a remarkable improvement to the efficiency of the model, in fact it would likely slow down training and inference when memory does in fact fit in the GPU. The point about trying many of random seeds to get convergence makes me wonder if the Neural GPU is worth its computational cost at all, when evaluated as means of learning algorithms that are already well understood (e.g. parsing and evaluating S-exprs). Consider spending all of the computational cycles that go into training one of these models (with the multiple seeds) on a traditional search through program space (e.g. sampling lisp programs or something). The notes on the curriculum strategies employed to get the presented results were interesting to read, as an indication of the lengths to which someone might have to go to train this sort of model, but it does leave this reviewer with the impression that despite the stated extensions of the Neural GPU model it remains unclear how useful it might be to practical problems.
assistant
Overall the paper has the feel of a status update by some of the best researchers in the field. The paper is very clear, the observations are interesting, but the remarks are scattered and don*t add up to a quantum of progress in the study of what can be done with the Neural GPU model. Minor remark on the use of the term RNN in Table 1: I found Table 1 confusing because several of the columns are for models that are technically RNNs, and use of RNNs for e.g. translation and word2vec highlight that RNNs can be characterized in terms of the length of their input sequence, the length of their input, and the sizes (per step) of their input, output, and working memories. Basic model question: How are inputs presented (each character 1-hot?) and outputs retrieved when there are e.g. 512 “filters” in the model ? If inputs and outputs are 1-hot encoded, and treated with the same filters as intermediate layers, then the intermediate activation functions should be interpretable as digits, and we should be able to interpret the filters as implementing a reliable e.g. multiplication-with-carry algorithm. Looking at the intermediate values may shed some light on why the usually-working models fail on e.g. the pathological cases identified in Table 3. The preliminary experiment on input alignment is interesting in two ways: the seeds for effective use of an attentional mechanism are there, but also, it suggests that the model is not presently dealing with general expression evaluation the way a correct algorithm should. The remarks in the abstract about improving the memory efficiency of Neural GPU seem overblown -- the paragraph at the top of page 6 describes the improvements as using tf.while_loop instead of unrolling the graph, and using swap_memory to use host memory when GPU memory runs short. These both seem like good practice, but not a remarkable improvement to the efficiency of the model, in fact it would likely slow down training and inference when memory does in fact fit in the GPU. The point about trying many of random seeds to get convergence makes me wonder if the Neural GPU is worth its computational cost at all, when evaluated as means of learning algorithms that are already well understood (e.g. parsing and evaluating S-exprs). Consider spending all of the computational cycles that go into training one of these models (with the multiple seeds) on a traditional search through program space (e.g. sampling lisp programs or something). The notes on the curriculum strategies employed to get the presented results were interesting to read, as an indication of the lengths to which someone might have to go to train this sort of model, but it does leave this reviewer with the impression that despite the stated extensions of the Neural GPU model it remains unclear how useful it might be to practical problems.