Scalable Model Selection for Belief Networks
We propose a scalable algorithm for model selection in sigmoid belief networks (SBNs), based on the factorized asymptotic Bayesian (FAB) framework. We derive the corresponding generalized factorized information criterion (gFIC) for the SBN, which is proven to be statistically consistent with the marginal log-likelihood. To capture the dependencies within hidden variables in SBNs, a recognition network is employed to model the variational distribution. The resulting algorithm, which we call FABIA, can simultaneously execute both model selection and inference by maximizing the lower bound of gFIC. On both synthetic and real data, our experiments suggest that FABIA, when compared to state-of-the-art algorithms for learning SBNs, (i) produces a more concise model, thus enabling faster testing; (ii) improves predictive performance; (iii) accelerates convergence; and (iv) prevents overfitting.
The authors propose a variational Bayes method for model selection in sigmoid belief networks. The method can eliminate nodes in the hidden layers of multilayer networks. The derivation of the criterion appears technically solid and a fair amount of experimental support for the good performance of the method is provided.
I have to say I'm no expert in this area, and I hope other reviewers can comment on the level of novelty.
detailed comments:
- p. 1, l. 21: "Model selection is here the task of selecting the number of layers [...]": I got the impression that the proposed algorithm only eliminates individual nodes, not entire layers. Please clarify.
- p. 2, ll. 76--77: Are there really no weights on the first layer? You forgot to define b.
- p. 5, ll. 173--175: If nodes with expected proportion of 1's very small can be eliminated, why doesn't the same hold for nodes with expected proportion of 0's equally small? "When the expectation is not exact, such as in the top layers, [...]": Please clarify. How can we tell, when the expectation is not exact? And do you really mean 'exact' or just 'accurate', etc. What is the precise rule to decide this.
- p. 6, ll. 226--227: Please explain what 10-5^2 and 25-15 means (is there a typo, or why would you write 5^2 instead of simply 25?).
- p. 6, l. 239: "Our performance metric is the variational lower bound of the test log-likelihood." Why use a variational bound as a performance metric? Sounds like variational techniques are an approach to derive a criterion, but using them also to measure performance sounds questionable. Should the evaluation be based on a score that is independent of the chosen approach and the made assumptions?
- references: please provide proper bibliographic information, not just "JMLR" or "NIPS". Add volume, page numbers, etc.