Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic needed to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn to add and subtract; and Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is to our knowledge the first arithmetic neural network component that can learn multiplication of a vector with a large hidden size. The two new components draw inspiration from a theoretical analysis of recent arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our results, compared with previous attempts, show that the NAU and NMU converges more consistently, have fewer parameters, learn faster, do not diverge with large hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small numbers.
DISCLAIMER: I reviewed a previous version of this paper at another venue.
This paper introduces Neural Addition Units (NAU) and Neural Multiplication Units (NMU), essentially redeveloped models of the Neural Arithmetic Logic Unit (NALU). The paper presents a strong case that the new models outperform NALUs in a few metrics: rate of convergence, learning speed, parameter number and model sparsity. The performance of NAU is better than NAC/NALU, as is the performance of NMU with a caveat that the presented NMU here cannot deal with division, though it can deal with negative numbers (as opposed to NALU).
What this paper excels at is a thorough theoretical and practical analysis of NALU’s issues and how the authors design the two new models to overcome these issues. The presented issues of NALU are numerous, including unstable optimization space, expectations of gradients converging to zero, the inability of NALUs gating to work as well as intended and its issues with division, and finally, the intended values of -1, 0, 1 in NALU do not get as close to these values as intended. 
The paper is easy to read, modulo a number of typos and admittedly some weirdly written sentences (see typos and minor issues later) and I would definitely recommend another iteration over the text to improve the issues with it as well as the style of writing. I am quite fond of the analysis and the informed design of the two new models, as well as the simplicity of the final models which are fairly close to the original models but have been shown both theoretically and practically that they work.
It is great to see that the paper improved since my last review and stands stronger on its results, but there are still a few issues with it that make me hesitant to fully accept the paper:
- The conclusion of the paper is biased towards the introduced models, but it should clearly define the limitations of these models wrt NALU/NAC
- The performance of NALu on multiplication is in stark contrast to the results in the original paper (Table 1). This should be commented in the paper why that is, as the original model presents no issues of NALU with multiplication, whereas this paper essentially says that they haven’t gotten a single model (out of 100 of them) to do multiplication.
- Could you explicitly comment on the paper why is the parameter sparsity such a sought-after quality of these models?
- You ‘assume an approximate discrete solution with parameters close to {1-, 0, 1} is important’. What do you have to back this assumption? Would it be possible to learn the arithmetic operations (and generalize) even with parameters different than those?
- Why did you introduce the product of the sequential MNIST experiment but did not presents results on the original sum / counting of digits? The change makes it hard to compare with the results in the original paper, and you do not present the reason why. This also makes me ask why didn't you compare to NALU on more tasks presented in the paper?
To conclude, this paper presents a well-done experimental and theoretical analysis of the issues of NALU and ways to fix it. Though the models presented outperform NALU, they still come with their own issues, namely they do not support division, and (admittedly, well corroborated with analysis) are not joined in a single, NALU-like model, that can learn multiple arithmetic operations. The paper does a great analysis of the models’ issues, with an experimental setup that highlights these issues, however, it does that on only one task from the original paper, and a(n insufficiently justified) modification of another one (multiplication of MNIST digits)---it does not extensively test these models on the same experimental setup as the original paper does.
Typos and smaller issues:
- Throughout the text you say that NMU supports large hidden input sizes? Why hidden??
- Figure 4 is identical to figure in D.2
- Repetition that E[z] = 0 is a desired property in 2.2, 2.3, 2.4
- In Related work, binary representation -> one-hot representation
- Found empirically in () - remove parentheses and see
- increasing the hidden size -> hidden vector size?
- NAU and NMU converges/learns/doesobtains -> converge/learn/do/obtain
- hard learn -> hard to learn ?
- NAU and NMU ...and improves -> improve
- Table 1 show -> shows
- Caption Table 1: Shows the - quite unusual caption (treating Table 1 as part of the sentence), would suggest to rephrase (e.g. Comparison/results of … on the … task). Similarly for Table 2...and Figure 3
- experiemnts -> experiments
- To analyse the impact of each improvements….. - this sentence is missing a chunk of it, or To should be replaced by We
- Allows NAC_+ to be -> remove be
- can be express as -> expressed
- The Neural Arithmetic Expression Calculator () propose learning - one might read this as the model proposes, not the authors / paper / citation propose…(also combine or combines in the next line)
- That the NMU models works -> model works? models work?
- We choice the -> we choose the 
- hindre -> hinder
- C.5 seperate -> separate
- There’s a number of typos in the appendix
- convergence the first -> convergence of the first?
- Where the purpose is to fit an unknown function -> I think a more appropriate statement would hint at an often overparameterization in practice done when fitting a(n unknown) function