Equi-normalization of Neural Networks
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18.
 The authors propose a new weight re-parameterization technique called Equi-normalization (ENorm) inspired by the Sinkhorn-Knopp algorithm. The authors show that the proposed method preserve functionally equivalent property in respect of the output of the functions (Linear, Conv, and Max-Pool) and show also that ENorm converges to the global optimum through the optimization. The experimental results show that ENorm performs better than baseline methods on CIFAR-10 and ImageNet datasets. pros) (+) The authors provide a theoretical ground. (+) The theoretical analysis of the convergence of the proposed algorithm is well provided. (+) The computational overhead reduced by the proposed method compared with BN and GN looks good. cons) (-) There is no comparison with other weight reparameterization methods such as Weight Normalization, Normalization propagation, Instance Normalization, or Layer Normalization. (-) The evidence why functionally equivalence is connected to the performance or generalization ability is not clarified. (-) The experimental results cannot consistently show the effectiveness of the proposed method in test accuracy. In Table 4, the proposed method outperforms BN, but In Table 2 and 3, BN is mostly better than the proposed method. (-) The batch size shown in Table 2 and 3 may be intended to show the batch-independent property of the proposed method, but BN is also doing well in those tables. Therefore, Table 2 and 3 are not adequate to show the batch-independent property. (-) The proposed method should evaluate with deeper networks (e.g., ResNet-50, ResNet101, or DenseNet-169) to support the superiority over BN and GN. (-) Adjusting c does not seem to be promising. In Table 2 and 3, ENorm-1 is better than ENorm-1.2, and also in Table 4, only the result of ENorm-1 is provided. The authors should do a parameter study with c to make all the experiments more convincing. Comments) - The experimental settings are not consistent. The authors should provide the reason why they set those settings or should include some studies about the parameters (for example about the paramter c). - Section 3.7 is not clear to me. How*s the performance going on when adjusting c < 1? - It is better for the authors to provide the Sinkhorn-Knopp algorithm (SK algorithm), which gave them an inspiration for this work, for better readability. - Why eq.(4) is necessary? For iterative optimization? If so, the authors should incorporate a detailed explanation about this in the corresponding section. - The authors should provide a detailed description of the parameter c. It is not clear why c is necessary, and please make sure the overall derivation does not need to be modified due to the emergence of c. - It seems that the authors could compact the paper by highlighting key ideas. - Typo: Annex A (on p.5). The paper is written well and provides a sound theoretical analysis to show the main idea, but unfortunately, the experimental results do not seem to support the effectiveness of the proposed method.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Equi-normalization of Neural Networks
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose a new weight re-parameterization technique called Equi-normalization (ENorm) inspired by the Sinkhorn-Knopp algorithm. The authors show that the proposed method preserve functionally equivalent property in respect of the output of the functions (Linear, Conv, and Max-Pool) and show also that ENorm converges to the global optimum through the optimization. The experimental results show that ENorm performs better than baseline methods on CIFAR-10 and ImageNet datasets. pros) (+) The authors provide a theoretical ground. (+) The theoretical analysis of the convergence of the proposed algorithm is well provided. (+) The computational overhead reduced by the proposed method compared with BN and GN looks good. cons) (-) There is no comparison with other weight reparameterization methods such as Weight Normalization, Normalization propagation, Instance Normalization, or Layer Normalization. (-) The evidence why functionally equivalence is connected to the performance or generalization ability is not clarified. (-) The experimental results cannot consistently show the effectiveness of the proposed method in test accuracy. In Table 4, the proposed method outperforms BN, but In Table 2 and 3, BN is mostly better than the proposed method. (-) The batch size shown in Table 2 and 3 may be intended to show the batch-independent property of the proposed method, but BN is also doing well in those tables. Therefore, Table 2 and 3 are not adequate to show the batch-independent property. (-) The proposed method should evaluate with deeper networks (e.g., ResNet-50, ResNet101, or DenseNet-169) to support the superiority over BN and GN. (-) Adjusting c does not seem to be promising. In Table 2 and 3, ENorm-1 is better than ENorm-1.2, and also in Table 4, only the result of ENorm-1 is provided. The authors should do a parameter study with c to make all the experiments more convincing. Comments) - The experimental settings are not consistent. The authors should provide the reason why they set those settings or should include some studies about the parameters (for example about the paramter c). - Section 3.7 is not clear to me. How*s the performance going on when adjusting c < 1? - It is better for the authors to provide the Sinkhorn-Knopp algorithm (SK algorithm), which gave them an inspiration for this work, for better readability. - Why eq.(4) is necessary? For iterative optimization? If so, the authors should incorporate a detailed explanation about this in the corresponding section. - The authors should provide a detailed description of the parameter c. It is not clear why c is necessary, and please make sure the overall derivation does not need to be modified due to the emergence of c. - It seems that the authors could compact the paper by highlighting key ideas. - Typo: Annex A (on p.5). The paper is written well and provides a sound theoretical analysis to show the main idea, but unfortunately, the experimental results do not seem to support the effectiveness of the proposed method.
assistant
 The authors propose a new weight re-parameterization technique called Equi-normalization (ENorm) inspired by the Sinkhorn-Knopp algorithm. The authors show that the proposed method preserve functionally equivalent property in respect of the output of the functions (Linear, Conv, and Max-Pool) and show also that ENorm converges to the global optimum through the optimization. The experimental results show that ENorm performs better than baseline methods on CIFAR-10 and ImageNet datasets. pros) (+) The authors provide a theoretical ground. (+) The theoretical analysis of the convergence of the proposed algorithm is well provided. (+) The computational overhead reduced by the proposed method compared with BN and GN looks good. cons) (-) There is no comparison with other weight reparameterization methods such as Weight Normalization, Normalization propagation, Instance Normalization, or Layer Normalization. (-) The evidence why functionally equivalence is connected to the performance or generalization ability is not clarified. (-) The experimental results cannot consistently show the effectiveness of the proposed method in test accuracy. In Table 4, the proposed method outperforms BN, but In Table 2 and 3, BN is mostly better than the proposed method. (-) The batch size shown in Table 2 and 3 may be intended to show the batch-independent property of the proposed method, but BN is also doing well in those tables. Therefore, Table 2 and 3 are not adequate to show the batch-independent property. (-) The proposed method should evaluate with deeper networks (e.g., ResNet-50, ResNet101, or DenseNet-169) to support the superiority over BN and GN. (-) Adjusting c does not seem to be promising. In Table 2 and 3, ENorm-1 is better than ENorm-1.2, and also in Table 4, only the result of ENorm-1 is provided. The authors should do a parameter study with c to make all the experiments more convincing. Comments) - The experimental settings are not consistent. The authors should provide the reason why they set those settings or should include some studies about the parameters (for example about the paramter c). - Section 3.7 is not clear to me. How*s the performance going on when adjusting c < 1? - It is better for the authors to provide the Sinkhorn-Knopp algorithm (SK algorithm), which gave them an inspiration for this work, for better readability. - Why eq. (4) is necessary? For iterative optimization? If so, the authors should incorporate a detailed explanation about this in the corresponding section. - The authors should provide a detailed description of the parameter c. It is not clear why c is necessary, and please make sure the overall derivation does not need to be modified due to the emergence of c. - It seems that the authors could compact the paper by highlighting key ideas. - Typo: Annex A (on p.5). The paper is written well and provides a sound theoretical analysis to show the main idea, but unfortunately, the experimental results do not seem to support the effectiveness of the proposed method.