Function Space Particle Optimization for Bayesian Neural Networks
While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and over-parameterized nature. To address this issue, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have been proposed. These methods directly optimize a set of particles to approximate the target posterior. However, their application to BNNs often yields sub-optimal performance, as such methods have a particular failure mode on over-parameterized models. In this paper, we propose to solve this issue by performing particle optimization directly in the space of regression functions. We demonstrate through extensive experiments that our method successfully overcomes this issue, and outperforms strong baselines in a variety of tasks including prediction, defense against adversarial examples, and reinforcement learning.
 This paper considers particle optimization variational inference methods for Bayesian neural networks. To avoid degeneracies which arise when these algorithms are applied to the weight space posterior, the authors consider applying the approach in the function space. A heuristic motivation is given for their algorithm and it seems to have good empirical performance. I find the paper well-motivated and the suggested algorithm original and interesting. As the authors mention at one point the derivation is rather heuristic, so much depends on the empirical assessment of their approach. I was wondering if it was worthwhile to include an architecture search of some kind in the empirical comparisons in the examples? This is because if wider than needed hidden layers are used this will worsen some of the degeneracies of the weight space posterior which could make the weight space algorithms perform worse. Also the authors use a Gaussian process approximation in part of their algorithm and wide hidden layers make that approximation more reasonable and may advantage their approach for that reason too. The authors discuss in Appendix B other approaches to improving weight space POVI. I wonder also if parameter constraints would be helpful for improving the performance of the weight space methods, such as order constraints on the hidden layer biases for example to remove at least some of the sources of unidentifiability. The authors talk in the introduction about the difficulties of exploring a complex high-dimensional posterior, the curse of dimensionality, and the limitations of current variational families but only 20 points are used to represent the posterior in the examples. Are many more particles required to obtain good performance in more complex models and does the approach scale well in terms of its computational requirements in that sense?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Function Space Particle Optimization for Bayesian Neural Networks
While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and over-parameterized nature. To address this issue, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have been proposed. These methods directly optimize a set of particles to approximate the target posterior. However, their application to BNNs often yields sub-optimal performance, as such methods have a particular failure mode on over-parameterized models. In this paper, we propose to solve this issue by performing particle optimization directly in the space of regression functions. We demonstrate through extensive experiments that our method successfully overcomes this issue, and outperforms strong baselines in a variety of tasks including prediction, defense against adversarial examples, and reinforcement learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper considers particle optimization variational inference methods for Bayesian neural networks. To avoid degeneracies which arise when these algorithms are applied to the weight space posterior, the authors consider applying the approach in the function space. A heuristic motivation is given for their algorithm and it seems to have good empirical performance. I find the paper well-motivated and the suggested algorithm original and interesting. As the authors mention at one point the derivation is rather heuristic, so much depends on the empirical assessment of their approach. I was wondering if it was worthwhile to include an architecture search of some kind in the empirical comparisons in the examples? This is because if wider than needed hidden layers are used this will worsen some of the degeneracies of the weight space posterior which could make the weight space algorithms perform worse. Also the authors use a Gaussian process approximation in part of their algorithm and wide hidden layers make that approximation more reasonable and may advantage their approach for that reason too. The authors discuss in Appendix B other approaches to improving weight space POVI. I wonder also if parameter constraints would be helpful for improving the performance of the weight space methods, such as order constraints on the hidden layer biases for example to remove at least some of the sources of unidentifiability. The authors talk in the introduction about the difficulties of exploring a complex high-dimensional posterior, the curse of dimensionality, and the limitations of current variational families but only 20 points are used to represent the posterior in the examples. Are many more particles required to obtain good performance in more complex models and does the approach scale well in terms of its computational requirements in that sense?
assistant
 This paper considers particle optimization variational inference methods for Bayesian neural networks. To avoid degeneracies which arise when these algorithms are applied to the weight space posterior, the authors consider applying the approach in the function space. A heuristic motivation is given for their algorithm and it seems to have good empirical performance. I find the paper well-motivated and the suggested algorithm original and interesting. As the authors mention at one point the derivation is rather heuristic, so much depends on the empirical assessment of their approach. I was wondering if it was worthwhile to include an architecture search of some kind in the empirical comparisons in the examples? This is because if wider than needed hidden layers are used this will worsen some of the degeneracies of the weight space posterior which could make the weight space algorithms perform worse. Also the authors use a Gaussian process approximation in part of their algorithm and wide hidden layers make that approximation more reasonable and may advantage their approach for that reason too. The authors discuss in Appendix B other approaches to improving weight space POVI. I wonder also if parameter constraints would be helpful for improving the performance of the weight space methods, such as order constraints on the hidden layer biases for example to remove at least some of the sources of unidentifiability. The authors talk in the introduction about the difficulties of exploring a complex high-dimensional posterior, the curse of dimensionality, and the limitations of current variational families but only 20 points are used to represent the posterior in the examples. Are many more particles required to obtain good performance in more complex models and does the approach scale well in terms of its computational requirements in that sense?