Noisy Machines: Understanding noisy neural networks and enhancing robustness to analog hardware errors using distillation
The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference. In particular, analog computing hardware has been heavily motivated specifically for accelerating neural networks, based on either electronic, optical or photonic devices, which may well achieve lower power consumption than conventional digital electronics. However, these proposed analog accelerators suffer from the intrinsic noise generated by their physical components, which makes it challenging to achieve high accuracy on deep neural networks. Hence, for successful deployment on analog accelerators, it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights, which is a somewhat new challenge in machine learning. In this paper, we advance the understanding of noisy neural networks. We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output. To combat this, we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks, which is demonstrated experimentally across different networks and datasets, including ImageNet. Our method achieves models with as much as 2X greater noise tolerance compared with the previous best attempts, which is a significant step towards making analog hardware practical for deep learning.
I started reading the paper with high hopes. The abstract and the introduction were set up nicely and I was quite intrigued to see a theoretical analysis and practical implementation of a neural network using analogue hardware. However, as I read through the paper carefully multiple times, I realized that the expository opening description fails to live up to the standard it sets. To be more specific, I did not enjoy the elaborate description of noisy analogue conductances as the noise models analyzed in the paper are not tested on any of such devices. The noise model introduced is fairly simplistic and arguably, the real-world systems are much more complex compared to such simplistic assumptions. The authors could have presented the paper as an analysis of knowledge distillation in neural network training. Even if the paper were presented that way, I would have doubted its chance of acceptance due to the incrementality of the theoretical contribution. All in all, I believe this is a very promising direction to invest, but the paper is not quite ready for ICLR.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Noisy Machines: Understanding noisy neural networks and enhancing robustness to analog hardware errors using distillation
The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference. In particular, analog computing hardware has been heavily motivated specifically for accelerating neural networks, based on either electronic, optical or photonic devices, which may well achieve lower power consumption than conventional digital electronics. However, these proposed analog accelerators suffer from the intrinsic noise generated by their physical components, which makes it challenging to achieve high accuracy on deep neural networks. Hence, for successful deployment on analog accelerators, it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights, which is a somewhat new challenge in machine learning. In this paper, we advance the understanding of noisy neural networks. We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output. To combat this, we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks, which is demonstrated experimentally across different networks and datasets, including ImageNet. Our method achieves models with as much as 2X greater noise tolerance compared with the previous best attempts, which is a significant step towards making analog hardware practical for deep learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



I started reading the paper with high hopes. The abstract and the introduction were set up nicely and I was quite intrigued to see a theoretical analysis and practical implementation of a neural network using analogue hardware. However, as I read through the paper carefully multiple times, I realized that the expository opening description fails to live up to the standard it sets. To be more specific, I did not enjoy the elaborate description of noisy analogue conductances as the noise models analyzed in the paper are not tested on any of such devices. The noise model introduced is fairly simplistic and arguably, the real-world systems are much more complex compared to such simplistic assumptions. The authors could have presented the paper as an analysis of knowledge distillation in neural network training. Even if the paper were presented that way, I would have doubted its chance of acceptance due to the incrementality of the theoretical contribution. All in all, I believe this is a very promising direction to invest, but the paper is not quite ready for ICLR.
assistant
I started reading the paper with high hopes. The abstract and the introduction were set up nicely and I was quite intrigued to see a theoretical analysis and practical implementation of a neural network using analogue hardware. However, as I read through the paper carefully multiple times, I realized that the expository opening description fails to live up to the standard it sets. To be more specific, I did not enjoy the elaborate description of noisy analogue conductances as the noise models analyzed in the paper are not tested on any of such devices. The noise model introduced is fairly simplistic and arguably, the real-world systems are much more complex compared to such simplistic assumptions. The authors could have presented the paper as an analysis of knowledge distillation in neural network training. Even if the paper were presented that way, I would have doubted its chance of acceptance due to the incrementality of the theoretical contribution. All in all, I believe this is a very promising direction to invest, but the paper is not quite ready for ICLR.