Learning Infinite RBMs with Frank-Wolfe
In this work, we propose an infinite restricted Boltzmann machine (RBM), whose maximum likelihood estimation (MLE) corresponds to a constrained convex optimization. We consider the Frank-Wolfe algorithm to solve the program, which provides a sparse solution that can be interpreted as inserting a hidden unit at each iteration, so that the optimization process takes the form of a sequence of finite models of increasing complexity. As a side benefit, this can be used to easily and efficiently identify an appropriate number of hidden units during the optimization. The resulting model can also be used as an initialization for typical state-of-the-art RBM training algorithms such as contrastive divergence, leading to models with consistently higher test likelihood than random initialization.
This paper describes a greedy method for adding hidden nodes to a restricted Boltzmann machine (RBM).  The method is derived by taking the marginal likelihood formula for the RBM, which involves a discrete sum of softplus functions using the weight vectors of the hidden units, and relaxing it to take an expectation with respect to an arbitrary distribution over weight vectors.  Optimizing this distribution using a method similar to functional gradient boosting yields the greedy algorithm for adding nodes.  The experimental results show that the resulting RBM benefits from further tuning using a more standard learning algorithm such as contrastive divergence, but the proposed algorithm is useful for finding the optimal number of hidden units, and provides a better-than-random initialization point for the RBM weights.
This paper is well-written and addresses the important practical issue of choosing the number of hidden nodes in an RBM.  The application of the Frank-Wolfe algorithm is not particularly novel, given the large set of related work that the paper cites.  But the paper makes the key technical move of replacing the discrete sum over hidden nodes in the marginal likelihood function with an arbitrary distribution over weight vectors.  This opens the path to applying Frank-Wolfe in a functional gradient setting.  I found this move interesting and novel (although there may be related work I don't know about); it might suggest similar approaches for other undirected models besides RBMs.  My main question is whether there are other techniques or heuristics that could be included for comparison in the experiments, besides the basic technique of training with various numbers of hidden nodes and various random initializations.  For instance, would it be feasible to compare to the ordered RBM approach of Cote and Larochelle?  Minor comments: * p. 1, line 37: "a efficient" should be "an efficient". * p. 4, line 140: "the second two terms in the gradient enforces".  Should be "enforce". * p. 5, line 169: "one can simply initialize...and continues".  Should be "continue". * p. 7, line 209: "CD-10".  Would be helpful to say what the 10 means here. * p. 7, line 218: "We randomly initialize the CD algorithm 5 times and select the best one".  The best one on the validation set, not the test set, right?  Would be good to make this explicit. * p. 7, line 227: "We evaluate the test likelihood of the model in FW after adding every 20 hidden units, and perform early stopping...".  The rest of this sentence suggests that the early stopping criterion is based just on the training and validation sets, not the test set, but the beginning of the sentence suggests that the test set is somehow involved.  It would be clearer to make these two separate sentences. * p. 8, Figure 2: I'm slightly color-blind and find it very hard to distinguish between the magenta line and the red line in these graphs.  Maybe use cyan instead of magenta, and use different marker shapes for the three series. * p. 8, line 254: "to higher test likelihood".  A verb is missing here; maybe "to achieve higher test likelihood".  == Update After Author Feedback == I appreciate the authors' responses to my comments.  It's interesting to hear that the iRBM tends to learn many more hidden units than the proposed method.  How does the accuracy compare between the two approaches?  I hope it will be feasible to include some comparison in the final version.