In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.
*Summary*
The paper applies variational inference (VI) with the 'reparameterisation' trick for Bayesian recurrent neural networks (BRNNs). The paper first considers the "Bayes by Backprop" approach of Blundell et al. (2015) and then modifies the BRNN model with a hierarchical prior over the network parameters, which then requires a hierarchical variational approximation with a simple linear recognition model. Several experiments demonstrate the quality of the prediction and the uncertainty over dropout.  
*Originality + significance*
To my knowledge, there is no other previous work on VI with the reparameterisation trick for BRNNs. However, one could say that this paper is, on careful examination, an application of reparameterisation gradient VI for a specific application. 
Nevertheless, the parameterisation of the conditional variational distribution q(\theta | \phi, (x, y)) using recognition model is interesting and could be useful in other models. However, this has not been tested or concretely shown in this paper. The idea of modifying the model by introducing variables to obtain a looser bound which can accommodate a richer variational family is also not new, see: hierarchical variational model (Ranganath et al., 2016) for example. 
*Clarity*
The paper is, in general, well-written. However, the presentation in 4 is hard to follow. I would prefer if appendix A3 was moved up front -- in this case, it would make it clear that the model is modified to contain \phi, a variational approximation over both \theta and \phi is needed, and a q that couples \theta, \phi and and the gradient of the log likelihood term wrt \phi is chosen. 
Additional comments:
Why is the variational approximation called "sharpened"?
At test time, normal VI just uses the fixed q(\theta) after training. It's not clear to me how prediction is done when using 'posterior sharpening' -- how is q(\theta | \phi, x) in eqs. 19-20 parameterised? The first paragraph of page 5 uses q(\theta | \phi, (x, y)), but y is not known at test time.
What is C in eq. 9?
This comment "variational typically underestimate the uncertainty in the posterior...whereas expectation propagation methods are mode averaging and so tend to overestimate uncertainty..." is not precise. EP can do mode averaging as well as mode seeking, depending on the underlying and approximate factor graphs. In the Bayesian neural network setting when the likelihood is factorised point-wise and there is one factor for each likelihood, EP is just as mode-seeking as variational. On the other hand, variational methods can avoid modes too, see the mixture of Gaussians example in the "Two problems with variational EM... " paper by Turner and Sahani (2010).
There are also many hyperparameters that need to be chosen -- what would happen if these are optimised using the free-energy? Was there any KL reweighting scheduling as done in the original BBB paper? 
What is the significance of the difference between BBB and BBB with sharpening in the language modelling task? Was sharpening used in the image caption generation task?
What is the computational complexity of BBB with posterior sharpening? Twice that BBB? If this is the case, would BBB get to the same performance if we optimise it for longer? Would be interesting to see the time/accuracy frontier.