We show that dropout training is best understood as performing MAP estimation concurrently for a family of conditional models whose objectives are themselves lower bounded by the original dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with tighter and higher lower bounds than the fully stochastic dropout objective. We argue that since the deterministic subvariant's bound is equal to its objective, and the highest amongst these models, the predominant view of it as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective.
This paper studies the problem of making predictions with a model trained using dropout. Authors try to provide a theoretical foundation for using dropout when making predictions. For this purpose, they show that when using dropout training we are maximizing a common lower bound on the objectives of a family of models, including most of the previously used methods for prediction with dropout. 
I find that the paper addresses a relevant problem and try to apply a novel approach. But, in general, I find the paper is not easy to follow and to grasp the main ideas. 
Here I detail my main concerns:
1. This is one of my main concerns. The contraposition between the geometric and the average model. I don't like this contraposition. The average model is just the standard marginalization operation over the weights, $p(y|x) = \int p(y|x,w)p(w|\Theta)dw$. This is the natural solution for the prediction problem to the problem if we accept the generative model given in Eq (3). 
In the case of the variational dropout, we depart from the same generative model, but we employ an approximation. It is the variational approximation the one that induces the geometric mean provided in eq (6). I.e. if we want to compute the posterior over the label y* for a sample x*, after training, we should compute the associated lower bound
$\ln p(y*|x*) >= E_q[\ln p(y*|x*,w)] - KL(q|p)$
In this case, q(w) = p(w|\Theta), as stated Eq (3) and in the corresponding equation provided in page 2 (the q(w) is not learnt because it only depends on the dropout rate, while the $\Theta$ are learnt by maximum log-likelihood and do not have a $q$ associated).  This gives rise to the geometric mean approximation provided in Eq (6).  I.e. the geometric mean prediction is simply the result of using a variational approximation at prediction time.   
My problem here is that authors employ convoluted arguments to introduce this geometric mean prediction and the average prediction, without making the connection discussed above. 
3. Section 3.3 and 3.4 introduces new arguments for modifying the dropout rate (and the alpha) parameter at test time. But, again, I find the arguments convoluted. We consider the dropout rate a hyper-parameter of the model, the standard learning theory tells us to fix the parameters with the training data and evaluate them later when making predictions. Why should we use different dropout rates at training and testing? Authors arguments about the tightness of the bound of Eq (8) and Eq(9). are not convincing to me. 
So, I don't find authors provide convincing answers to the raised questions at the beginning of the paper about the use of dropout when making predictions. 
Minor comments:
1. The generative model for Variational dropout is the same than the generative model for the "conditional model", eq. (3). 
2. In Eq. (7) authors are defining the weighted power mean. I think it would be clearer to directly introduce the weighted power mean instead of the standard power mean in Section 3.2.
3. Section 3.3. I find some parts are difficult to understand. "suppose we pick a base model from the power mean family and have a continuum of subvariants with gradually reduced variance in their predictions but the same expectation." Later, I can understand authors are referring to the possibility of reducing the dropout rate.