While neural network models achieve impressive classification accuracy across different tasks, they can suffer from poor calibration of their probabilistic predictions. A Bayesian perspective has recently suggested that dropout, a regularization strategy popularly used during training, can be employed to obtain better probabilistic predictions at test time (Gal & Ghahramani, 2016a). However, empirical results so far have not been encouraging, particularly with convolutional networks. In this paper, through the lens of ensemble learning, we associate this unsatisfactory performance with the correlation between the models sampled with dropout. Motivated by this, we explore the use of various structured dropout techniques to promote model diversity and improve the quality of probabilistic predictions. We also propose an omnibus dropout strategy that combines various structured dropout methods. Using the SVHN, CIFAR-10 and CIFAR-100 datasets, we empirically demonstrate the superior performance of omnibus dropout relative to several widely used strong baselines in addition to regular dropout. Lastly, we show the merit of omnibus dropout in a Bayesian active learning application.
PAPER SUMMARY: The paper argues that (ensembles of models with different types of dropout applied to each model) perform better than (ensembles in which the same type of dropout is applied to each model). They attribute this to increasing model diversity in the former case, and experimentally validate their claims.
MAJOR COMMENTS: 
1. Motivation Unclear:
- Notational issues in (2): MSE(h_t | x) involves y, but y is not specified in the definition. As a result, later E_{x}[…] is evaluated disregarding the dependence on y, which is ambiguous. 
- Derivation of the second equality in (2) is not obvious, and needs a detailed proof. Notational issues exist in switching between H and h.
- The above is used to argue that “the more diverse the models, the better performance achieved”. It is unclear how this follows from (2).
2. Unclear / Imprecise Writing
- Before Sec. 3.4:  “This is because higher Dropout rates lead to smaller effective network capacities” Needs reference. 
- “Dropout uncertainty can be obtained sequentially”. Unclear what this means.
3. The entire proposed method is described in one line, “we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods”. It is very unclear as to what the authors mean by combination.
4. Experiments
- It seems that Omnibus dropout leads to a moderate “diversity”, and improves ensemble performance only for SVHN, out of the three datasets tested in Figure 4. For SVHN, the improvement is 0.1% accuracy, which is within the standard error (0.1). Hence, it seems that the proposed method provides no performance improvement.
- A Similar statement is true for the other metrics (NLL, Brier Score, ECE). 
Score [Scale of 1-10]:
3: Reject
The paper needs either more convincing experiments demonstrating the claims or theoretical analysis explaining the behavior for small networks. The paper needs to be rewritten to improve presentation, and state motivation, problem statement, and contributions clearly.