Semi-Implicit Back Propagation
Neural network has attracted great attention for a long time and many researchers are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.
The paper introduces a novel algorithm for computing update directions for neural network*s weights. The algorithm consists of the modified backpropagation procedure where a layer*s error is computed using implicitly-updated weights. The proposed idea is interesting, but its presentation and evaluation could be significantly improved. First, it is not very clear what motivates the exact form of Semi-implicit BP. Second, I find the notation a bit cumbersome, especially intermediate ^{k+1/2} updates. I also suspect that eq. 9 contains an error, probably the l.h.s. of the first part should be delta_i^{k+1}? Algorithm 1 is not very helpful because only the forward pass is explained in detail and a reader must refer to the main text to understand the backward pass. The experimental evaluation also raises a number of questions. 1) Why did authors chose only one value of the learning rate and the lambda hyperparameter? A more appropriate comparison would require slightly more extensive hyperparameter search as it may well be that ProxBP would work better with different values. 2) It is also unclear if 5 CG iterations is enough to solve the intermediate problem. Also all convergence guarantees are only provided for the exact implicit update, so at least one should ensure it is computed well enough. 3) Isn*t it suspucious that ProxBP performed so bad compared to other methods on MNIST? 4) Should not the set of baselines include more advanced optimizers such as RMSProp, Adam etc? They don*t seem to add more computational burden than Semi-implicit BP. I also think it is worth discussing/investigating if the obtained update directions can be used in other gradient-based optimizers instead of pure gradients and if it can have any advantages. Overall, it does not feel to me that the paper is ready for publication.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Semi-Implicit Back Propagation
Neural network has attracted great attention for a long time and many researchers are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper introduces a novel algorithm for computing update directions for neural network*s weights. The algorithm consists of the modified backpropagation procedure where a layer*s error is computed using implicitly-updated weights. The proposed idea is interesting, but its presentation and evaluation could be significantly improved. First, it is not very clear what motivates the exact form of Semi-implicit BP. Second, I find the notation a bit cumbersome, especially intermediate ^{k+1/2} updates. I also suspect that eq. 9 contains an error, probably the l.h.s. of the first part should be delta_i^{k+1}? Algorithm 1 is not very helpful because only the forward pass is explained in detail and a reader must refer to the main text to understand the backward pass. The experimental evaluation also raises a number of questions. 1) Why did authors chose only one value of the learning rate and the lambda hyperparameter? A more appropriate comparison would require slightly more extensive hyperparameter search as it may well be that ProxBP would work better with different values. 2) It is also unclear if 5 CG iterations is enough to solve the intermediate problem. Also all convergence guarantees are only provided for the exact implicit update, so at least one should ensure it is computed well enough. 3) Isn*t it suspucious that ProxBP performed so bad compared to other methods on MNIST? 4) Should not the set of baselines include more advanced optimizers such as RMSProp, Adam etc? They don*t seem to add more computational burden than Semi-implicit BP. I also think it is worth discussing/investigating if the obtained update directions can be used in other gradient-based optimizers instead of pure gradients and if it can have any advantages. Overall, it does not feel to me that the paper is ready for publication.
assistant
The paper introduces a novel algorithm for computing update directions for neural network*s weights. The algorithm consists of the modified backpropagation procedure where a layer*s error is computed using implicitly-updated weights. The proposed idea is interesting, but its presentation and evaluation could be significantly improved. First, it is not very clear what motivates the exact form of Semi-implicit BP. Second, I find the notation a bit cumbersome, especially intermediate ^{k+1/2} updates. I also suspect that eq. 9 contains an error, probably the l.h.s. of the first part should be delta_i^{k+1}? Algorithm 1 is not very helpful because only the forward pass is explained in detail and a reader must refer to the main text to understand the backward pass. The experimental evaluation also raises a number of questions. 1) Why did authors chose only one value of the learning rate and the lambda hyperparameter? A more appropriate comparison would require slightly more extensive hyperparameter search as it may well be that ProxBP would work better with different values. 2) It is also unclear if 5 CG iterations is enough to solve the intermediate problem. Also all convergence guarantees are only provided for the exact implicit update, so at least one should ensure it is computed well enough. 3) Isn*t it suspucious that ProxBP performed so bad compared to other methods on MNIST? 4) Should not the set of baselines include more advanced optimizers such as RMSProp, Adam etc? They don*t seem to add more computational burden than Semi-implicit BP. I also think it is worth discussing/investigating if the obtained update directions can be used in other gradient-based optimizers instead of pure gradients and if it can have any advantages. Overall, it does not feel to me that the paper is ready for publication.