Estimating Learnability in the Sublinear Data Regime
We consider the problem of estimating how well a model class is capable of fitting a distribution of labeled data. We show that it is possible to accurately estimate this "learnability" even when given an amount of data that is too small to reliably learn any accurate model. Our first result applies to the setting where the data is drawn from a d-dimensional distribution with isotropic covariance, and the label of each datapoint is an arbitrary noisy function of the datapoint. In this setting, we show that with O( √ d) samples, one can accurately estimate the fraction of the variance of the label that can be explained via the best linear function of the data. For comparison, even if the labels are noiseless linear functions of the data, a sample size linear in the dimension, d, is required to learn any function correlated with the underlying model. Our estimation approach also applies to the setting where the data distribution has an (unknown) arbitrary covariance matrix, allowing these techniques to be applied to settings where the model class consists of a linear function applied to a nonlinear embedding of the data. In this setting we give a consistent estimator of the fraction of explainable variance that uses o(d) samples. Finally, our techniques also extend to the setting of binary classification, where we obtain analogous results under the logistic model, for estimating the classification accuracy of the best linear classifier. We demonstrate the practical viability of our approaches on synthetic and real data. This ability to estimate the explanatory value of a set of features (or dataset), even in the regime in which there is too little data to realize that explanatory value, may be relevant to the scientific and industrial settings for which data collection is expensive and there are many potentially relevant feature sets that could be collected.
When I started reading the paper, I actually liked it. But when I got to page 5, I was very disappointed and annoyed. It might not be intentional, but it felt almost deceptive that the authors did not mention very relevant prior art till then and made it look like many of the previous literature as their own contribution.  Authors claim that they propose an algorithm that uses O(sqrt(d)) samples for estimating the variance of noise.  In fact, after a read of this paper and previous papers, what I got was: O(sqrt(d)) was essentially known before (for most cases) and new results are in fact much weaker.   Dicker et al: Maximum Likelihood for Variance Estimation in High-Dimensional Linear Models (http://www.stat.rutgers.edu/home/ldicker/papers/variance.pdf), showed that O(sqrt(d)) samples are sufficient for learning sublinearity when additional noise (eta) is Gaussian i.i.d. and data points X are Gaussian and either have identity covariance or any general covariance (Lemma 2 and proposition 1 respectively). Further, for the convergence proof, the properties of Gaussianity they used were only that the moments are bounded. Hence their results and proofs more or less extend to non-Gaussian cases with similar moment bounds.  What the authors showed:  a. Extend the results of Dicker et al (with a very similar algorithm) to non-Gaussian but isotropic and bounded moments. But (mostly) the exact proof of Lemma 2 in Dicker et al holds for any isotropic distributions with bounded moments already.  b. Theorem 1: They propose a weaker version of the result when the X is not from a Gaussian and is non-isotropic. Here the result is O(d^(1-g(epsilon)), which is much weaker than sqrt(d). Further the convergence rate is weak 1/poly(log(d/n)) as opposed to the polynomial convergence rates of Dicker et al. The dependence of epsilon is also bad and super linear. (exponential in 1/epsilon in certain cases).  c. Extend isotropic results to the agnostic case, but here again, I am not sure what is the novelty compared to Lemma 2 in Dicker et al.  Also authors did not compare Dicker et al algorithms with the “new” one in experiments.  I request the authors to  a. Exactly state what is new in Algorithm 1 compared to the algorithm of Dicker et al. b. Why are Theorem 1 results much weaker than the Proposition 1 results of Dicker et al.? c. What is the novelty in Proposition 1 and Theorem 2 compared to the proof of Lemma 2 in Dicker et al. d. State the convergence rate of the algorithm (Theorem 1).  e. If the results are different than Dicker et al, include their algorithms in your experimental plots and compare experimentally which one works better.