Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
Recently deep neural networks have received considerable attention due to their ability to extract and represent high-level abstractions in data sets. Deep neural networks such as fully-connected and convolutional neural networks have shown excellent performance on a wide range of recognition and classification tasks. However, their hardware implementations currently suffer from large silicon area and high power consumption due to the their high degree of complexity. The power/energy consumption of neural networks is dominated by memory accesses, the majority of which occur in fully-connected networks. In fact, they contain most of the deep neural network parameters. In this paper, we propose sparsely-connected networks, by showing that the number of connections in fully-connected networks can be reduced by up to 90% while improving the accuracy performance on three popular datasets (MNIST, CIFAR10 and SVHN). We then propose an efficient hardware architecture based on linear-feedback shift registers to reduce the memory requirements of the proposed sparsely-connected networks. The proposed architecture can save up to 90% of memory compared to the conventional implementations of fully-connected neural networks. Moreover, implementation results show up to 84% reduction in the energy consumption of a single neuron of the proposed sparsely-connected networks compared to a single neuron of fully-connected neural networks.
From my original comments: The results looks good but the baselines proposed are quite bad. For instance in the table 2 *Misclassification rate for a 784-1024-1024-1024-10 * the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see *significant* compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers... In CIFAR-10 experiments, i do not understand why *Sparsely-Connected 90% + Single-Precision Floating-Point* is worse than *Sparsely-Connected 90% + BinaryConnect*. So it is better to use binary than float. Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations). Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision. In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines. ---- The authors reply still does not convince me. I still think that the same technique should be applied on more challenging scenarios.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
Recently deep neural networks have received considerable attention due to their ability to extract and represent high-level abstractions in data sets. Deep neural networks such as fully-connected and convolutional neural networks have shown excellent performance on a wide range of recognition and classification tasks. However, their hardware implementations currently suffer from large silicon area and high power consumption due to the their high degree of complexity. The power/energy consumption of neural networks is dominated by memory accesses, the majority of which occur in fully-connected networks. In fact, they contain most of the deep neural network parameters. In this paper, we propose sparsely-connected networks, by showing that the number of connections in fully-connected networks can be reduced by up to 90% while improving the accuracy performance on three popular datasets (MNIST, CIFAR10 and SVHN). We then propose an efficient hardware architecture based on linear-feedback shift registers to reduce the memory requirements of the proposed sparsely-connected networks. The proposed architecture can save up to 90% of memory compared to the conventional implementations of fully-connected neural networks. Moreover, implementation results show up to 84% reduction in the energy consumption of a single neuron of the proposed sparsely-connected networks compared to a single neuron of fully-connected neural networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



From my original comments: The results looks good but the baselines proposed are quite bad. For instance in the table 2 *Misclassification rate for a 784-1024-1024-1024-10 * the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see *significant* compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers... In CIFAR-10 experiments, i do not understand why *Sparsely-Connected 90% + Single-Precision Floating-Point* is worse than *Sparsely-Connected 90% + BinaryConnect*. So it is better to use binary than float. Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations). Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision. In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines. ---- The authors reply still does not convince me. I still think that the same technique should be applied on more challenging scenarios.
assistant
From my original comments: The results looks good but the baselines proposed are quite bad. For instance in the table 2 *Misclassification rate for a 784-1024-1024-1024-10 * the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see *significant* compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers... In CIFAR-10 experiments, i do not understand why *Sparsely-Connected 90% + Single-Precision Floating-Point* is worse than *Sparsely-Connected 90% + BinaryConnect*. So it is better to use binary than float. Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations). Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision. In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines. ---- The authors reply still does not convince me. I still think that the same technique should be applied on more challenging scenarios.