In this paper, we extend the Bayesian neural network to functional Bayesian neural network with functional Monte Carlo methods that use the samples of functionals instead of samples of networks' parameters for inference to overcome the curse of dimensionality for uncertainty quantification. Based on the previous work on Riemannian Langevin dynamics, we propose the stochastic gradient functional Riemannian dynamics for training functional Bayesian neural network. We show the effectiveness and efficiency of our proposed approach with various experiments.
The authors propose an approximate MCMC method for sampling a posterior distribution of weights in a Bayesian neural network.  They claim that existing MCMC methods are limited by poor scaling with dimensionality of the weights, and they propose a method inspired by HMC on finite-dimensional approximations of measures on an infinite-dimensional Hilbert space (Beskos et al, 2011).  In short, the idea is to use a low dimensional approximation to the parameters (i.e. weights) of the neural network, representing them instead as a weighted combination of basis functions in neural network parameter space.  Then the authors propose to use HMC on this lower dimensional representation.  While the idea is intriguing, there are a number of flaws in the presentation, notational inconsistencies, and missing experiments that prohibit acceptance in the current form.
The authors define a functional, f: \theta -> [0, 1], that maps neural network parameters \theta to the unit interval.  They claim that this function defines a probability distribution on \theta, but this not warranted.  First, \theta is a continuous random variable and its probability density need not be bounded above by one; second, the authors have made no constraints on f actually being normalized.  
The second flaw is that the authors equate a posterior on f given the data with a posterior on the parameters \theta themselves.  Cf. Eq 4 and paragraph above.  There is a big difference between a posterior on parameters and a posterior on distributions over parameters.   Moreover, Eq. 5 doesn't make sense: there is only one posterior f; there are no samples of the posterior. 
The third problem appears in the start of Section 3, where the authors now call the posterior U(theta) instead of f.  They make a finite approximation of posterior U(\theta) = \sum_i \lambda_i u_i, which is inconsistent with Beskos et al.  I believe the authors intend to use a low dimensional approximation to \theta rather than its posterior U(\theta).  For example, if \theta = \sum_i \lambda_i u_i for fixed basis functions u_i, then you can approximate a posterior on \theta with a posterior on \lambda.
The fourth, and most important problem, is that the basis functions u_i are never defined.  How are these chosen? Beskos et al use the eigenfunctions of the Gaussian base measure \pi_0, but no such measure exists here.  Moreover, this choice will have a substantial impact on the approximation quality. 
There are more inconsistencies and notational problems throughout the paper.  Section 4.1 begins with a mean field approximation that seems out of place.  Section 3 clearly states that the posterior on theta is approximated with a posterior on lambda, and this cannot factorize over the dimensions of theta.  Finally, the authors again confuse the posterior on weights with a posterior on distributions of weights in Eq 11.   \tilde{U} is introduced as a function of lambda in Eq 14 and then called with f in line 4 of Alg. 1.  These two types are not interchangeable. 
These inconsistencies cast doubt on the subsequent experiments.  Assuming the algorithm is correct, a fundamental experiment is still missing. 
To justify this approach, the authors should show how the posterior approximation quality varies as a function of the size of the low dimensional approximation, D.
I reiterate that the idea of approximating the posterior distribution over neural network weights with a posterior distribution over a lower dimensional representation of weights is interesting.  Unfortunately, the abundance of errors in presentation cloud the positive contributions of this paper.