Learning SMaLL Predictors
We introduce a new framework for learning in severely resource-constrained settings. Our technique delicately amalgamates the representational richness of multiple linear predictors with the sparsity of Boolean relaxations, and thereby yields classifiers that are compact, interpretable, and accurate. We provide a rigorous formalism of the learning problem, and establish fast convergence of the ensuing algorithm via relaxation to a minimax saddle point objective. We supplement the theoretical foundations of our work with an extensive empirical evaluation.
Summary of paper: The paper addresses the problem of learning in resource constrained setting using multiple linear predictors (max of linear predictors like DNF). The problem of learning k-DNFs (a subset of multiple linear predictord) is known to be hard however to solve this non-convex problem, they use various relaxations to formulate this as a minmax saddle point objective which they solve using standard mirror-prox algorithm. The paper then goes on to show that the approach is fruitful in practice through experiments.  Comments for authors: In the introduction (line 28-31), you seem to sugest that you can solve the problem of k-DNFs however you do not actually give any guarantees for the final learnt hypothesis that your algorithm outputs, please make that clear.  Requiring a different weight vector for each positive example, this seems very costly and potentially avoidable. It would be good to discuss how the clustering affects this, probably show a plot for different cluster sizes vs performance.  Overall comments: Using Multiple linear predictors as the base hypothesis class is a good direction as it is a richer class of functions compared to standard sparse linear regressors (LASSO). Viewing the problem as a minmax saddle point problem is an interesting take and I find the use of the boolean constraint matrix novel. As is, the algorithm is not efficient (dimension of weight vector can be of the order of the number of samples) and they do not give any theoretical guarantees for the hypothesis returned by their algorithm. However, the experimental results do seem to suggest that in practice this approach gives a compact sparse representation with low computational overhead. Overall, the paper is well written, provides a useful framework for solving a common problem and from a practical standpoint gives gains over most standard methods (not always large) while giving a handle on the exact sparsity required in the final hypothesis.