Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights
Multilayer Neural Networks (MNNs) are commonly trained using gradient descent-based methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expectation Propagation (EP). We show how an EP based approach can also be used to train deterministic MNNs. Specifically, we approximate the posterior of the weights given the data using a "mean-field" factorized distribution, in an online setting. Using online EP and the central limit theorem we find an analytical approximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs. Despite a different origin, the resulting algorithm, Expectation BackPropagation (EBP), is very similar to BP in form and efficiency. However, it has several additional advantages: (1) Training is parameter-free, given initial conditions (prior) and the MNN architecture. This is useful for large-scale problems, where parameter tuning is a major challenge. (2) The weights can be restricted to have discrete values. This is especially useful for implementing trained MNNs in precision limited hardware chips, thus improving their speed and energy efficiency by several orders of magnitude. We test the EBP algorithm numerically in eight binary text classification tasks. In all tasks, EBP outperforms: (1) standard BP with the optimal constant learning rate (2) previously reported state of the art. Interestingly, EBP-trained MNNs with binary weights usually perform better than MNNs with continuous (real) weights -if we average the MNN output using the inferred posterior.
The paper introduces a back propagation algorithm which instead of computing the gradient information, updates beliefs regarding posterior distributions of model parameters.  In doing so the learning algorithm relies on a mean-field approximation + the gaussian assumption (with the diagonal covariance matrix). They call this approach “expectation backdrop” (EBP).Overall,  I find some aspects of this work quite interesting. My questions / concerns / comments:1) The paper needs to provide a better discussion / comparison to the previous work on Bayesian learning of NNs ( see e.g., Radford Neal’s and David MacKay’s work from mid-90s + more recent work, e.g., on Langevin dynamics, ...). There have been a lot of work, and almost nothing is discussed here.2) The introduction of the EBP algorithm is motivated by arguing that the standard BP algorithm is not capable of learning NNs with binary weights which, according to the authors, much more efficient from the computational perspective at test time (however, one would need larger networks then?).   The EBP method is indeed capable of learning binary NNs in a fairly effective way. The catch is though that using binary models directly (i.e. the MAP solution) at test time directly does not seem to lead to competitive results (see column “Binary EBP-D” in Table 1: the error rate is  >= doubled for 4 datasets out of 7). What seems to work well is Bayesian model averaging with the binary model (“Binary EBP-P”) but this is again a ‘continuous’ computation which is at least as expensive as the computation with a normal NN.  At the very least the authors need to clarify this point, currently, as the binary version is the key motivation of this research (see the first paragraph of the paper).3) The notation in the model section needs to be cleaned up a bit. E.g., I was slightly confused with v vs. \nu (which is actually never formally introduced). The appendix was helpful but the paper needs to be understandable on its own.4) It would be interesting to see (empirically) how well this approach generalizes to deeper architecture (only models with a single hidden layers are considered in the paper).5) Since the argument is efficiently, I’d like to see a bit more discussion of this aspect in the paper (in the experimental section, e.g.). The approach is much more complex than standard BP (from the  implementation standpoint), and results even of the EBP-P version are not so obviously better than the ones for BP, so it would be nice to have more details to convince the readers. (However, the appendix does include some asymptotic analyses).