Neural Arithmetic Logic Units
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates. We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors. Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images. In contrast to conventional architectures, we obtain substantially better generalization both inside and outside of the range of numerical values encountered during training, often extrapolating orders of magnitude beyond trained numerical ranges.
The paper proposes a method to extrapolate numbers outside the training range in neural networks. It proposes two models in order to solve this task. The neural accumulator that transforms a linear layer to output additions or subtractions and the neural arithmethic logic unit which can learn more complex mathematical functions.   The tackled problem of extrapolation is an interesting one as neural networks generally perform not so well in such tasks. The paper itself is well motivated in the introduction and the example which compares different existing structures against each other adds additional clarity that this task should be studied.   The method is interesting, novel  and sound. It is well presented through good explanations and the diagrams in figure 2.   The large experiment section further supports the proposed approach. It is compared in different problem settings against many different baselines and even against each other. The promise as well as the limitations are discussed.   The  presentation of the idea overall was well done. However, it does have some gaps. A short background section, for example, could have introduced the reader to extrapolation as a problem and could have reiterated the most important neural network mechanisms as later a deeper knowledge is expected. Furthermore, the mathematical notation (M) used in line 58 onwards is never formally introduced.   Towards the end the authors mention that their work could also be seen as a different principled approach of designing neural networks in that it biases towards certain outcomes. the paper could have discussed this in more detail as this seems wider applicable than to only extrapolation problems and, therefore, more relevant to more readers.    Overall, the paper presented its main contributions clearly and showed in a large result section the importance as well as the success of their method. While the more general underlying principle would have been interesting to explore and a background section might have added additional clarity the paper's novel ideas and impressive results  still outline a significant contribution.