Ensemble Distribution Distillation
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of Ensemble Distribution Distillation (EnD^2) - distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD^2 enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD^2 based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD^2 are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD^2 can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.
Summary: Ensembles of probabilistic models (e.g. for classification) provide measures of both data uncertainty (i.e. how uncertain each model is on average) and model uncertainty (i.e. how much the models disagree in their predictions). When naively distilling an ensemble to a single model, the ability to decompose total uncertainty into data uncertainty and model uncertainty is lost. This paper describes a method for ensemble distillation that retains both types of uncertainty in the case of classification. The idea is to train a prior network (i.e. a conditional Dirichlet distribution) to model the distribution of categorical probabilities within the ensemble. Both total uncertainty and data uncertainty can then be computed analytically from the prior network. Pros: The paper considers an interesting problem, and makes a clear contribution towards addressing it. The paper motivates the problem well, and explains the contribution and the method*s limitations clearly. The proposed method is simple, but well motivated, sound, and well explained. The paper is very well written and easy to read. I particularly appreciated the toy experiment in section 4 and the visualization in figure 3, which showcase clearly what the method does. The experiments are thorough, and the results are discussed fairly and objectively. Cons: The scope of the paper and the method is limited to the problem of probabilistic classification. However, one could have a more general ensemble of conditional or unconditional distributions. The method could in principle be applied to this setting, by having a hypernetwork learn the distribution of the distributional parameters of the models in the ensemble (the method presented in the paper is a special case of this, where the hypernetwork is a prior network and the distributional parameters are categorical probabilities). However, it*s not clear that the method would scale to models with arbitrary distributional parameters. I would suggest that the paper make it clear from the beginning that the scope is probabilistic classification, and at the end discuss the extent to which this is a limitation, and how the method could potentially be extended to other kinds of ensembles. The prior network used is essentially a conditional Dirichlet distribution, which (as the paper clearly acknowledges) may not always be flexible enough. A more flexible prior network could be a mixture of Dirichlet distributions, where the mixing coefficients and the parameters of each mixture component would be functions of the input, similarly to mixture density networks (http://publications.aston.ac.uk/id/eprint/373/) but with Dirichets instead of Gaussians. I believe that equations (9) and (10) would still be tractable in that case, as it*s tractable to compute expectations under mixtures if the expectations under mixture components are tractable. One limitation of the approach is that the prior network may not give accurate predictions for inputs it hasn*t been trained on (as the paper discusses and section 4 clearly demonstrates). It*s not clear how this problem can be overcome in general, and further research may be needed in that direction. Some of the results in Table 4 are puzzling (as the paper also acknowledges). In particular, the EnD model should be able to retain the performance of the ensemble when using total uncertainty but it doesn*t. Also, using knowledge uncertainty doesn*t always seem to be better than using total uncertainty, which to some extent defeats the purpose of the method (at least in this particular example). It would be good to investigate further these results. In any case, I appreciate that the paper acknowledges these results, but avoids unjustified speculation about what may be causing them. Decision: Overall, this is good work and I*m happy to recommend acceptance. There are some limitations to the method, but these can be seen as motivation for future work. Minor suggestions for improvement: This older work is relevant and could be cited (but there is no obligation to do so). On compressing ensembles: - Model compression, https://dl.acm.org/citation.cfm?id=1150464 - Compact approximations to Bayesian predictive distributions, https://dl.acm.org/citation.cfm?id=1102457 - Distilling model knowledge, https://arxiv.org/abs/1510.02437 On information-theoretic measures for decomposing total uncertainty as in eq. (4): - Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, https://arxiv.org/abs/1710.07283 *[Knowledge uncertainty] arises when the test input comes from a different distribution than the one that generated the training data* This is only one way knowledge uncertainty can arise, it could also arise when the test input comes from the same distribution that generated training data, but there aren*t enough training data available. Some parentheses are dangling at the bottom of page 2, top of page 3 and middle of page 4. In figure 1, it*d be good to make clear in the caption that the figure is illustrating the simplex of categorical probabilities. The last paragraph of section 2 uses the term *posterior* repeatedly to refer to P(y|x, 	heta) which is confusing. I would call P(y|x, 	heta) the *model prediction* or something like that. Eq. (5) should be without the negative sign I think. In section 3, I would use a different symbol (e.g. phi) to denote the parameters of the prior network, to clearly distinguish them from the parameters of the models in the ensemble. *Optimization of the KL-divergence between distributions with limited non-zero common support is particularly difficult* Some more evidence in support of this claim is needed I think; either explain why or provide a reference that does. In section 4, the text says that EnD has *a minor degradation in performance* but table 1 seems to show otherwise. Also, the results of EnD in table 1 and table 2 are different, which makes me think there may be a typo somewhere. Making the references have consistent format and correct capitalization (e.g. DNNs, Bayesian) would make the paper look even more polished.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Ensemble Distribution Distillation
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of Ensemble Distribution Distillation (EnD^2) - distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD^2 enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD^2 based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD^2 are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD^2 can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: Ensembles of probabilistic models (e.g. for classification) provide measures of both data uncertainty (i.e. how uncertain each model is on average) and model uncertainty (i.e. how much the models disagree in their predictions). When naively distilling an ensemble to a single model, the ability to decompose total uncertainty into data uncertainty and model uncertainty is lost. This paper describes a method for ensemble distillation that retains both types of uncertainty in the case of classification. The idea is to train a prior network (i.e. a conditional Dirichlet distribution) to model the distribution of categorical probabilities within the ensemble. Both total uncertainty and data uncertainty can then be computed analytically from the prior network. Pros: The paper considers an interesting problem, and makes a clear contribution towards addressing it. The paper motivates the problem well, and explains the contribution and the method*s limitations clearly. The proposed method is simple, but well motivated, sound, and well explained. The paper is very well written and easy to read. I particularly appreciated the toy experiment in section 4 and the visualization in figure 3, which showcase clearly what the method does. The experiments are thorough, and the results are discussed fairly and objectively. Cons: The scope of the paper and the method is limited to the problem of probabilistic classification. However, one could have a more general ensemble of conditional or unconditional distributions. The method could in principle be applied to this setting, by having a hypernetwork learn the distribution of the distributional parameters of the models in the ensemble (the method presented in the paper is a special case of this, where the hypernetwork is a prior network and the distributional parameters are categorical probabilities). However, it*s not clear that the method would scale to models with arbitrary distributional parameters. I would suggest that the paper make it clear from the beginning that the scope is probabilistic classification, and at the end discuss the extent to which this is a limitation, and how the method could potentially be extended to other kinds of ensembles. The prior network used is essentially a conditional Dirichlet distribution, which (as the paper clearly acknowledges) may not always be flexible enough. A more flexible prior network could be a mixture of Dirichlet distributions, where the mixing coefficients and the parameters of each mixture component would be functions of the input, similarly to mixture density networks (http://publications.aston.ac.uk/id/eprint/373/) but with Dirichets instead of Gaussians. I believe that equations (9) and (10) would still be tractable in that case, as it*s tractable to compute expectations under mixtures if the expectations under mixture components are tractable. One limitation of the approach is that the prior network may not give accurate predictions for inputs it hasn*t been trained on (as the paper discusses and section 4 clearly demonstrates). It*s not clear how this problem can be overcome in general, and further research may be needed in that direction. Some of the results in Table 4 are puzzling (as the paper also acknowledges). In particular, the EnD model should be able to retain the performance of the ensemble when using total uncertainty but it doesn*t. Also, using knowledge uncertainty doesn*t always seem to be better than using total uncertainty, which to some extent defeats the purpose of the method (at least in this particular example). It would be good to investigate further these results. In any case, I appreciate that the paper acknowledges these results, but avoids unjustified speculation about what may be causing them. Decision: Overall, this is good work and I*m happy to recommend acceptance. There are some limitations to the method, but these can be seen as motivation for future work. Minor suggestions for improvement: This older work is relevant and could be cited (but there is no obligation to do so). On compressing ensembles: - Model compression, https://dl.acm.org/citation.cfm?id=1150464 - Compact approximations to Bayesian predictive distributions, https://dl.acm.org/citation.cfm?id=1102457 - Distilling model knowledge, https://arxiv.org/abs/1510.02437 On information-theoretic measures for decomposing total uncertainty as in eq. (4): - Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, https://arxiv.org/abs/1710.07283 *[Knowledge uncertainty] arises when the test input comes from a different distribution than the one that generated the training data* This is only one way knowledge uncertainty can arise, it could also arise when the test input comes from the same distribution that generated training data, but there aren*t enough training data available. Some parentheses are dangling at the bottom of page 2, top of page 3 and middle of page 4. In figure 1, it*d be good to make clear in the caption that the figure is illustrating the simplex of categorical probabilities. The last paragraph of section 2 uses the term *posterior* repeatedly to refer to P(y|x, 	heta) which is confusing. I would call P(y|x, 	heta) the *model prediction* or something like that. Eq. (5) should be without the negative sign I think. In section 3, I would use a different symbol (e.g. phi) to denote the parameters of the prior network, to clearly distinguish them from the parameters of the models in the ensemble. *Optimization of the KL-divergence between distributions with limited non-zero common support is particularly difficult* Some more evidence in support of this claim is needed I think; either explain why or provide a reference that does. In section 4, the text says that EnD has *a minor degradation in performance* but table 1 seems to show otherwise. Also, the results of EnD in table 1 and table 2 are different, which makes me think there may be a typo somewhere. Making the references have consistent format and correct capitalization (e.g. DNNs, Bayesian) would make the paper look even more polished.
assistant
Summary: Ensembles of probabilistic models (e.g. for classification) provide measures of both data uncertainty (i.e. how uncertain each model is on average) and model uncertainty (i.e. how much the models disagree in their predictions). When naively distilling an ensemble to a single model, the ability to decompose total uncertainty into data uncertainty and model uncertainty is lost. This paper describes a method for ensemble distillation that retains both types of uncertainty in the case of classification. The idea is to train a prior network (i.e. a conditional Dirichlet distribution) to model the distribution of categorical probabilities within the ensemble. Both total uncertainty and data uncertainty can then be computed analytically from the prior network. Pros: The paper considers an interesting problem, and makes a clear contribution towards addressing it. The paper motivates the problem well, and explains the contribution and the method*s limitations clearly. The proposed method is simple, but well motivated, sound, and well explained. The paper is very well written and easy to read. I particularly appreciated the toy experiment in section 4 and the visualization in figure 3, which showcase clearly what the method does. The experiments are thorough, and the results are discussed fairly and objectively. Cons: The scope of the paper and the method is limited to the problem of probabilistic classification. However, one could have a more general ensemble of conditional or unconditional distributions. The method could in principle be applied to this setting, by having a hypernetwork learn the distribution of the distributional parameters of the models in the ensemble (the method presented in the paper is a special case of this, where the hypernetwork is a prior network and the distributional parameters are categorical probabilities). However, it*s not clear that the method would scale to models with arbitrary distributional parameters. I would suggest that the paper make it clear from the beginning that the scope is probabilistic classification, and at the end discuss the extent to which this is a limitation, and how the method could potentially be extended to other kinds of ensembles. The prior network used is essentially a conditional Dirichlet distribution, which (as the paper clearly acknowledges) may not always be flexible enough. A more flexible prior network could be a mixture of Dirichlet distributions, where the mixing coefficients and the parameters of each mixture component would be functions of the input, similarly to mixture density networks (http://publications.aston.ac.uk/id/eprint/373/) but with Dirichets instead of Gaussians. I believe that equations (9) and (10) would still be tractable in that case, as it*s tractable to compute expectations under mixtures if the expectations under mixture components are tractable. One limitation of the approach is that the prior network may not give accurate predictions for inputs it hasn*t been trained on (as the paper discusses and section 4 clearly demonstrates). It*s not clear how this problem can be overcome in general, and further research may be needed in that direction. Some of the results in Table 4 are puzzling (as the paper also acknowledges). In particular, the EnD model should be able to retain the performance of the ensemble when using total uncertainty but it doesn*t. Also, using knowledge uncertainty doesn*t always seem to be better than using total uncertainty, which to some extent defeats the purpose of the method (at least in this particular example). It would be good to investigate further these results. In any case, I appreciate that the paper acknowledges these results, but avoids unjustified speculation about what may be causing them. Decision: Overall, this is good work and I*m happy to recommend acceptance. There are some limitations to the method, but these can be seen as motivation for future work. Minor suggestions for improvement: This older work is relevant and could be cited (but there is no obligation to do so). On compressing ensembles: - Model compression, https://dl.acm.org/citation.cfm?id=1150464 - Compact approximations to Bayesian predictive distributions, https://dl.acm.org/citation.cfm?id=1102457 - Distilling model knowledge, https://arxiv.org/abs/1510.02437 On information-theoretic measures for decomposing total uncertainty as in eq. (4): - Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, https://arxiv.org/abs/1710.07283 *[Knowledge uncertainty] arises when the test input comes from a different distribution than the one that generated the training data* This is only one way knowledge uncertainty can arise, it could also arise when the test input comes from the same distribution that generated training data, but there aren*t enough training data available. Some parentheses are dangling at the bottom of page 2, top of page 3 and middle of page 4. In figure 1, it*d be good to make clear in the caption that the figure is illustrating the simplex of categorical probabilities. The last paragraph of section 2 uses the term *posterior* repeatedly to refer to P(y|x, 	heta) which is confusing. I would call P(y|x, 	heta) the *model prediction* or something like that. Eq. (5) should be without the negative sign I think. In section 3, I would use a different symbol (e.g. phi) to denote the parameters of the prior network, to clearly distinguish them from the parameters of the models in the ensemble. *Optimization of the KL-divergence between distributions with limited non-zero common support is particularly difficult* Some more evidence in support of this claim is needed I think; either explain why or provide a reference that does. In section 4, the text says that EnD has *a minor degradation in performance* but table 1 seems to show otherwise. Also, the results of EnD in table 1 and table 2 are different, which makes me think there may be a typo somewhere. Making the references have consistent format and correct capitalization (e.g. DNNs, Bayesian) would make the paper look even more polished.