Neural Networks with Block Diagonal Inner Product Layers
Artificial neural networks have opened up a world of possibilities in data science and artificial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem. We make contributions to this issue by considering a modified version of the fully connected layer we call a block diagonal inner product layer. These modified layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups. This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers. Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries. This method condenses network storage and speeds up the run time without significant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efficiency.
 This is a mostly experimental paper which evaluates the capabilities of neural networks with weight matrices that are block diagonal. The authors describe two methods to obtain this structure: (1) enforced during training, (2) enforced through regularization and pruning. As a second contribution, the authors show experimentally that the random matrix theory can provide a good model of the spectral behavior of the weight matrix when it is large. However, the authors only conjecture as to the potential of this method without describing clear ways of approaching this subject, which somewhat lessens the strength of their argument. Quality: this paper is of good quality Clarity: this paper is clear, but would benefit from better figures and from tables to report the numerical results instead of inserting them into plain text. Originality: this paper introduces block diagonal matrices to structure the weights of a neural network. The idea of structured matrices in this context is not new, but the diagonal block structure appears to be. Significance: This paper is somewhat significant. PROS - A new approach to analyzing the behavior of weight matrices during learning - A new structure for weight matrices that provides good performance while reducing matrix storage requirements and speeding up forward and backward passes. CONS - Some of the figures are hard to read (in particular Fig 1 & 2 left) and would benefit from a better layout. - It would be valuable to see experiments on bigger datasets than only MNIST and CIFAR-10. - I understand that the main advantage of this method is the speedup; however, providing the final accuracy as a function of the nonzero entries for slower methods (e.g. the sparse pruning showed in Fig 1. a) would provide a more complete picture. Main questions: - Could you briefly comment on the training time in section 4.1? - Could you elaborate on the last sentence of section 4.1? - You state: "singular values of an IP layer behave according to the MP distribution even after 1000s of training iterations." Is this a known fact, or something that you observed empirically? In practice, how large must the weight matrix be to observe this behavior? Nitpicks: - I believe the term "fully connected" is more standard than "inner product" and would add clarity to the paper, but I may be mistaken.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Neural Networks with Block Diagonal Inner Product Layers
Artificial neural networks have opened up a world of possibilities in data science and artificial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem. We make contributions to this issue by considering a modified version of the fully connected layer we call a block diagonal inner product layer. These modified layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups. This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers. Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries. This method condenses network storage and speeds up the run time without significant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efficiency.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This is a mostly experimental paper which evaluates the capabilities of neural networks with weight matrices that are block diagonal. The authors describe two methods to obtain this structure: (1) enforced during training, (2) enforced through regularization and pruning. As a second contribution, the authors show experimentally that the random matrix theory can provide a good model of the spectral behavior of the weight matrix when it is large. However, the authors only conjecture as to the potential of this method without describing clear ways of approaching this subject, which somewhat lessens the strength of their argument. Quality: this paper is of good quality Clarity: this paper is clear, but would benefit from better figures and from tables to report the numerical results instead of inserting them into plain text. Originality: this paper introduces block diagonal matrices to structure the weights of a neural network. The idea of structured matrices in this context is not new, but the diagonal block structure appears to be. Significance: This paper is somewhat significant. PROS - A new approach to analyzing the behavior of weight matrices during learning - A new structure for weight matrices that provides good performance while reducing matrix storage requirements and speeding up forward and backward passes. CONS - Some of the figures are hard to read (in particular Fig 1 & 2 left) and would benefit from a better layout. - It would be valuable to see experiments on bigger datasets than only MNIST and CIFAR-10. - I understand that the main advantage of this method is the speedup; however, providing the final accuracy as a function of the nonzero entries for slower methods (e.g. the sparse pruning showed in Fig 1. a) would provide a more complete picture. Main questions: - Could you briefly comment on the training time in section 4.1? - Could you elaborate on the last sentence of section 4.1? - You state: "singular values of an IP layer behave according to the MP distribution even after 1000s of training iterations." Is this a known fact, or something that you observed empirically? In practice, how large must the weight matrix be to observe this behavior? Nitpicks: - I believe the term "fully connected" is more standard than "inner product" and would add clarity to the paper, but I may be mistaken.
assistant
 This is a mostly experimental paper which evaluates the capabilities of neural networks with weight matrices that are block diagonal. The authors describe two methods to obtain this structure: (1) enforced during training, (2) enforced through regularization and pruning. As a second contribution, the authors show experimentally that the random matrix theory can provide a good model of the spectral behavior of the weight matrix when it is large. However, the authors only conjecture as to the potential of this method without describing clear ways of approaching this subject, which somewhat lessens the strength of their argument. Quality: this paper is of good quality Clarity: this paper is clear, but would benefit from better figures and from tables to report the numerical results instead of inserting them into plain text. Originality: this paper introduces block diagonal matrices to structure the weights of a neural network. The idea of structured matrices in this context is not new, but the diagonal block structure appears to be. Significance: This paper is somewhat significant. PROS - A new approach to analyzing the behavior of weight matrices during learning - A new structure for weight matrices that provides good performance while reducing matrix storage requirements and speeding up forward and backward passes. CONS - Some of the figures are hard to read (in particular Fig 1 & 2 left) and would benefit from a better layout. - It would be valuable to see experiments on bigger datasets than only MNIST and CIFAR-10. - I understand that the main advantage of this method is the speedup; however, providing the final accuracy as a function of the nonzero entries for slower methods (e.g. the sparse pruning showed in Fig 1. a) would provide a more complete picture. Main questions: - Could you briefly comment on the training time in section 4.1? - Could you elaborate on the last sentence of section 4.1? - You state: "singular values of an IP layer behave according to the MP distribution even after 1000s of training iterations." Is this a known fact, or something that you observed empirically? In practice, how large must the weight matrix be to observe this behavior? Nitpicks: - I believe the term "fully connected" is more standard than "inner product" and would add clarity to the paper, but I may be mistaken.