We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresponding backpropagated gradient. Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam. Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW. As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms. Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.
In this work, the authors attempt to unify existing adaptive gradient methods under the Bayesian filtering framework with the dynamical prior.  In Ollivier, 2017, a framework is proposed to connect Bayesian filtering and natural gradient.  On the other hand,  in Khan et al., 2018. an approach is proposed to connect natural gradient and adaptive gradient methods.  The main contributions of this work are (1)  introducing a dynamical prior and (2) recovering RMSProp and Adam as special cases. 
However, the proposed dynamical prior is very similar to the fading memory technique used in Ollivier, 2017. (see Proposition 3 of Ollivier, 2017) 
Furthermore, the authors argue that this work recovers a root-mean-square form while Khan et al., 2018 recovers a different sum-square form. Unfortunately, the authors have to use a series of unnatural approximations to recover the root-mean-square form. In fact, as mentioned in Khan, 2017b  this proposed method without these approximations is also a mean-square form. (also see Eq (2.28-2.29) of Ollivier, 2017)
Since the authors mainly follow Ollivier, 2017 and make unnatural approximations,  the work has a limited impact.  To get a higher rating, the authors should clearly give justifications and insights of these approximations.
Detailed comments:
(1) On Page 1,  "The typical approach to Bayesian filtering, where we infer a distribution, ... jointly, forces us to use extremely strong, factorised approximations, and it is legitimate to worry that these strong approximations might meaningfully disrupt the ability of Bayesian filtering to give close-to-optimal updates.   ... we instead consider ... that incorporates factorisation into the problem setting, and therefore requires fewer approximations downstream. "
The proposed method is equivalent to jointly perform Kalman filtering with full-covariance with an additional diagonal-approximation step. This additional step might also meaningfully disrupt the ability of Bayesian filtering. Furthermore, such approximation ignores the off-diagonal terms in the low-rank approximation at Eq (8). 
Minor: You should use \approx at Eq (8) since a rank-1 approximation is used.  
(2) On page 2, "It has been noted that under specific circumstances, natural gradient is approximate Bayesian filtering (Ollivier, 2017), allowing us to link Bayesian filtering to the rich literature on natural gradients.  However, this only occurs when the dynamical prior in the Bayesian filtering problem has a specific form: the parameters being fixed over time (i.e.  arguably an online data, rather than a true Bayesian filtering setting)." 
The authors should comment the difference between the dynamical prior and the fading memory technique (see Proposition 3 of Ollivier, 2017) where at page 14 of Ollivier, 2017, Ollivier mentions that "this is equivalent ... or to the addition of an artificial process noise ... in the model".  I think Ollivier's idea is very similar to the dynamical prior used at Eq (1) of this submission.  Furthermore, the second-order Taylor expansion with a Fisher information-based estimation of Hessian (see the equation below Eq(1) of this submission) is exactly the same as Ollivier's Extended Kalman filter (see  Eq 2.25 at Lemma 9  and Lemma 10 of Ollivier, 2017).  The authors should cite Ollivier, 2017.
Minor: Eq (6) should be E_p [ - \nabla_z^2 \log p(d|z) ] = E_p  [ e e^T ], where "-", the negative sign is missing. Please see the definition of the Fisher information matrix.
 
(3) On page 2, "While there have been attempts to use natural gradients to recover the Adam or RMSprop root-mean-square form for the gradient normalizer, in practice a different sum-square form emerges (Khan & Lin, 2017; Khan et al., 2018). In contrast, we show that to recover the Adam or RMSprop form for the gradient normalizer." 
Khan et al., 2018 is a mean-square form for variational inference due to the entropy term of the variational distribution. (see Sec 3 and 5 of  Khan et al., 2018 and Khan, 2017b )
Unfortunately, the "root-mean-square form" does not appear naturally in this submission. In practice, the proposed update is also a mean-square form  (see Eq (2.28-2.29) of Ollivier, 2017 and Khan, 2017b) without a series of unnatural approximations used in this submission.
To justify these assumptions, the authors should explain when "the steady state posterior variance" (see sec 2.21) and  "a self-consistent solution" (see sec 7.1) achieve.  As far as I know, \sigma^2_t = \sigma^2_{t+1} in sec 2.2.1 only holds in the limit case when t-> \inifity.  Why does the equality hold at each time step t? The authors should give a justification or an intuition about these approximations since this paper is a theory paper. Please also see my next point.
(4) Section 7.1 is also confusing.
In sec 7.1, the authors assume that A \in O(\eta). However, A=\eta^2/(2\sigma^2) in sec 2.2 and A_{1,1} =  ( \eta_w^2+\eta^2 )/ (2\sigma^2) at Eq (14). In both cases, A can be \in O(\eta^2). This is very *critical* since the authors argue that O(\eta^3) can be neglected in sec 7.1.  The authors use this point to show that Adam is a special case. 
If A \in O(\eta^2), we know that "A \Sigma_{post}" \in O(\eta^3) should be neglected. At the last equation on page 10,  the authors do not neglect "A \Sigma_{post}". Why?  The authors should clarify this point to avoid doing *selective* neglection.  Again, the impact of this paper should be inspiring new adaptive methods.
The authors also mention that the second-order term in A is neglected in sec 7.2. Any justification? 
References
[1] Ollivier, Yann. "Online Natural Gradient as a Kalman Filter." arXiv preprint arXiv:1703.00209 (2017).
[2] Khan, Mohammad Emtiyaz, and Wu Lin. "Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models." arXiv preprint arXiv:1703.04265 (2017).
[3] Khan, Mohammad Emtiyaz, et al. "Vprop: Variational Inference using RMSprop." arXiv preprint arXiv:1712.01038 (2017b).
[4] Khan, Mohammad Emtiyaz, et al. "Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam" (2018)