Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models
Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models* likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio test akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, and complexity estimates.
This paper analyzes the peculiar case that deep generative models often assign a higher likelihood to other datasets than they were trained on. The running hypothesis here is, that input complexity plays a central role. Measuring a proxy for input complexity shows that it is tightly anticorrelated with likelihood and therefore seems to describe the trend well. A new OOD detection score is introduced based on these insights. The failure of Cifar-10 generative models to detect SVHN as OOD via thresholded likelihood is an interesting question. I also think that the conclusion of this work, that this behavior may be natural and a consequence of likelihood itself, seems very sensible. The overall message of the paper is very important and might help to avoid researchers wasting time on improving likelihood models for OOD detection. Instead, it seems a more sensible way to deal with this problem is to consider likelihood ratios. One concern is that a formal discussion if the likelihood is supposed to behave the way the authors describe, or if a better model might solve this eventually. I think it is already mentioned that empirically there is no reason to believe a better model would solve this, it would just be nice to have a theoretical statement here as well. Another concern is that even though the empirical results look quite promising, it would be good to stress-test the proposed score on more common OOD detection benchmarks against state-of-the-art methods to see if likelihood generative models with the proposed criterion are competitive there. All in all, I think this is a great paper, additional theoretical analysis and stronger empirical results would be helpful to increase the significance of the work. -------------- Post Rebuttal -------------- I would like to thank the authors for their interesting response. However, I will not change my score as the response did not change my opinion on the need for a more theoretical treatment of the statement, which is a serious (and potentially hard to resolve) shortcoming of the paper in my opinion.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models
Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models* likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio test akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, and complexity estimates.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper analyzes the peculiar case that deep generative models often assign a higher likelihood to other datasets than they were trained on. The running hypothesis here is, that input complexity plays a central role. Measuring a proxy for input complexity shows that it is tightly anticorrelated with likelihood and therefore seems to describe the trend well. A new OOD detection score is introduced based on these insights. The failure of Cifar-10 generative models to detect SVHN as OOD via thresholded likelihood is an interesting question. I also think that the conclusion of this work, that this behavior may be natural and a consequence of likelihood itself, seems very sensible. The overall message of the paper is very important and might help to avoid researchers wasting time on improving likelihood models for OOD detection. Instead, it seems a more sensible way to deal with this problem is to consider likelihood ratios. One concern is that a formal discussion if the likelihood is supposed to behave the way the authors describe, or if a better model might solve this eventually. I think it is already mentioned that empirically there is no reason to believe a better model would solve this, it would just be nice to have a theoretical statement here as well. Another concern is that even though the empirical results look quite promising, it would be good to stress-test the proposed score on more common OOD detection benchmarks against state-of-the-art methods to see if likelihood generative models with the proposed criterion are competitive there. All in all, I think this is a great paper, additional theoretical analysis and stronger empirical results would be helpful to increase the significance of the work. -------------- Post Rebuttal -------------- I would like to thank the authors for their interesting response. However, I will not change my score as the response did not change my opinion on the need for a more theoretical treatment of the statement, which is a serious (and potentially hard to resolve) shortcoming of the paper in my opinion.
assistant
This paper analyzes the peculiar case that deep generative models often assign a higher likelihood to other datasets than they were trained on. The running hypothesis here is, that input complexity plays a central role. Measuring a proxy for input complexity shows that it is tightly anticorrelated with likelihood and therefore seems to describe the trend well. A new OOD detection score is introduced based on these insights. The failure of Cifar-10 generative models to detect SVHN as OOD via thresholded likelihood is an interesting question. I also think that the conclusion of this work, that this behavior may be natural and a consequence of likelihood itself, seems very sensible. The overall message of the paper is very important and might help to avoid researchers wasting time on improving likelihood models for OOD detection. Instead, it seems a more sensible way to deal with this problem is to consider likelihood ratios. One concern is that a formal discussion if the likelihood is supposed to behave the way the authors describe, or if a better model might solve this eventually. I think it is already mentioned that empirically there is no reason to believe a better model would solve this, it would just be nice to have a theoretical statement here as well. Another concern is that even though the empirical results look quite promising, it would be good to stress-test the proposed score on more common OOD detection benchmarks against state-of-the-art methods to see if likelihood generative models with the proposed criterion are competitive there. All in all, I think this is a great paper, additional theoretical analysis and stronger empirical results would be helpful to increase the significance of the work. -------------- Post Rebuttal -------------- I would like to thank the authors for their interesting response. However, I will not change my score as the response did not change my opinion on the need for a more theoretical treatment of the statement, which is a serious (and potentially hard to resolve) shortcoming of the paper in my opinion.