The Limits of Learning with Missing Data
We study linear regression and classification in a setting where the learning algorithm is allowed to access only a limited number of attributes per example, known as the limited attribute observation model. In this well-studied model, we provide the first lower bounds giving a limit on the precision attainable by any algorithm for several variants of regression, notably linear regression with the absolute loss and the squared loss, as well as for classification with the hinge loss. We complement these lower bounds with a general purpose algorithm that gives an upper bound on the achievable precision limit in the setting of learning with missing data.
The manuscript provides lower bounds for regression (and classification) in the limited attributes observation model. In particular, the lower bounds tighten what is previously known for the squared loss in this setting, and novel lower bounds for absolute loss and hinge loss (in classification) are shown. An algorithm that achieves loss bound up to a certain precision limit is also provided. The results are sufficiently novel and the paper is well-written.
The authors provide lower bounds for regression and classification in the limited attributes observation model. Here, the learning algorithm has access to a subset of attributes, and we want to control the excess loss (over the bayes optimal). The manuscript provides lower bounds in this setting that improve over known bounds, and resolve some open questions in this setting.   The manuscript makes interesting contributions: the provided lower bounds tighten what is previously known for the squared loss in this setting, and for absolute loss and hinge loss (in classification), novel lower bounds are derived. Furthermore, an algorithm that achieves loss bound up to a certain precision limit is also provided; this matches lower-bound up-to a polynomial factor in dimensionality (in case of regression). The results and proof techniques are sufficiently novel and the paper is well-written. I recommend acceptance.  The results in classification setting, albeit, appear less interesting (there is an exponential gap between the lower bound and the guaranteed precision limits of the proposed algorithm for hinge loss). The authors do not comment on Theorem 5 after it is briefly stated in Section 2.1. In particular, the nature of the result in Theorem 5 is different from that in Theorems 1 and 3 (for regression), and seem weaker (existence of some epsilon vs actually quantifying the limits of precision). I would like the authors to comment on this aspect, and perhaps an exposition in the main paper as to the difficulty of stating a stronger result (if that is the case) would be useful.    (I've read the author response.)