Learning with Pseudo-Ensembles
We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout [9] in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We examine the relationship of pseudo-ensembles, which involve perturbation in model-space, to standard ensemble methods and existing notions of robustness, which focus on perturbation in observation-space. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of [19] into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.
This paper proposes the notion of a pseudo-ensemble, that of dependent child models ensembled together, which is an nice and unified way of several related techniques prominently including dropout. Then presented the pseudo ensemble variance regularizer and tested it with convincing empirical results. The presented technique seems much easier to apply to multi-layer, semi-supervised setting than the technique of [23].Some objections are that the notion of this pseudo-ensemble as an expectation is already in references [5] and [23], so the contribution of naming it and generalizing to multi-layer is not a significant leap. The authors made an effort of gaining more understanding by writing down the boosty and baggy forms, but nothing was really done with them and the connection to boosting is vague. Finally, while it is okay to introduce an regularizer and show that it is empirically successful, it would be nice if it corresponds to, or is approximating something that we understand. The structure of paper would have me believe that this is done, as sections 1-3 talks about a general framework, and section 4 should be testing it. Instead, section 4 introduces a new regularizer that has little to do with the previous sections.  I would have given the paper a higher score based on its empirical strength and the simplicity of the proposed regularizer compared to the alternatives, if not for the fancy presentation that made it harder to read but did not add much insights. Please explain in the response if I am wrong on this and I am willing to change my scores.Detailed comments:the formalism of f(x; \xi) is much more general than the situation the paper seek to deal with: that of subsampling child models through \xi. Without alluding to that \xi represents something like dropout, this formalism does not mean much. Unfortunately, this seems hard to remedy without making the notations more complicated, so perhaps it is okay.Line 106 "Note how the left objective moves the loss L inside the expectation over noise"...this contradicts the equation, where the loss L is outside the expectation over noiseEq 2): baggy PE does not seem to make sense to me, are you summing over i? or is one of the expectations supposed to be taken over i as well?While there is some intuitive resemblance (final prediction is made by the sum of weak learners, though the sum is weighted in boosting), it is unclear to me how these boosty PE concretely connect to boosting. The flavor of sequentially adding more classifiers is not there, nor is the boosting weights assigned to each example. Can you make this more concrete? If not, perhaps it would be good to say this is how you define boosty in the first place and the reader should not be looking for rigorous connections.In line 129-140, you compared a few existing approaches and say that they only deal with the input space. I do not think this is entirely true. My perspective on this:A: [5] Learning with MCFs: non-convex (convex in some cases) lower bound bymoving the expectation inside the log but not inside the exp.B: [23] adaptive regularization: delta method (second-order central limit)expansion (relies on 4th moments small). This has the advantage of giving an explicit regularizer that's interpretable. Non-convex.C: Wang and Manning, Fast Dropout: relies on central limittheorem and sum of noise converging to Gaussian. Non-convex and deals with model noise as well as input noise.D: Baldi and Sadowski, Understanding dropout paper, analyzes geometric mean, also deals with model noise. Eq 3): why is there a subscript on the variance notation? it is the same operation regardless of i (correction: not so in line 210: is it worth the subscript though). You should perhaps call \mathcal{V} scale-invariant variance penalty, so the reader does not think you are talking about the actual variance. The font helps, but you still said its variance. What is the motivation of this regularization method? can you show that this converges to [23] for linear models somehow? or can you provide more understanding by somehow deriving/approximate it instead of just defining it?