Neural network has attracted great attention for a long time and many researchers are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.
The work is based on the recent paper 'Proximal Backpropagation', Frerix et al., ICLR 2018, which views error back propagation as block coordinate gradient descent steps on a penalty functional comprised of activations and parameters.
Instead of taking proximal steps on the linear layers as in (Frerix et al., 2018), the authors also pull the non-linearity \sigma into the proximal steps. Another interesting deviation is the idea to consider the newly updated weights {W_i}^{k+1} when updating the activations F_i^{k+1} in the backward pass.
While potentially offering a faster convergence with respect to epochs, the nonlinear updates have two major drawbacks:
1) While there are preliminary theoretical results (fixed points of the method are critical points), it remains unclear whether the computed update is still a descent direction on the original energy.  While not crucial, such a result would be reassuring and might give further insights into the method.
2) Each update requires the solution of a nonconvex, nonlinear least squares problem which is prohibitively expensive to solve. Note that such nonlinear least squares updates are already proposed in (Carreira-Perpinan & Wang, 2014). When using ReLU activation, the non smoothness might be an issue for standard nonlinear least squares solvers such as Levenberg-Marquadt. 
Furthermore, the numerical results are unfortunately a bit discouraging. The experiments evaluate toy models on toy datasets and even there only a minor improvement with respect to epochs over SGD and Prox-BP is shown. Furthermore, the plots only consider epochs and not the running time. Due to the non-linear least squares problem, I assume that each epoch for the proposed method is way more costly. Therefore I consider the experimental evaluation too preliminary. A proper evaluation would require an implementation as an optimizer in state-of-the-art deep learning frameworks and a comparison with respect to running time to standard optimizers such as SGD with momentum or Adam on the GPU. 
The reported performances for MNIST are surprisingly poor. Note that vanilla SGD with momentum reaches ~98.6% test set performance on such an architecture, while the overall highest reported accuracy in this paper is 98.0%. This might be due to momentum, and it would be interesting whether the proposed method could be combined with momentum or other optimizers such as Adam as in (Frerix et al. 2018).
Overall, I don't see this a practical algorithm for training deep networks and there are few theoretical results. Therefore, I cannot recommend acceptance at this stage. 
To improve the paper, I would like to see an implementation of the method on the GPU in a recent deep learning framework and an evaluation on larger models / datasets. But I am doubtful this will reach competitive performance to standard optimizers. Also, it would be interesting to see how the precision in the inner nonlinear conjugate gradient solver effects outer convergence. It might be that the subproblem does not have to be solved with very high accuracy. 
Minor comments:
* Missing citation: 'Difference Target Propagation', https://arxiv.org/abs/1412.7525, studies a similar type of algorithm.