Sample Efficient Stochastic Gradient Iterative Hard Thresholding Method for Stochastic Sparse Linear Regression with Limited Attribute Observation
We develop new stochastic gradient methods for efficiently solving sparse linear regression in a partial attribute observation setting, where learners are only allowed to observe a fixed number of actively chosen attributes per example at training and prediction times. It is shown that the methods achieve essentially a sample complexity of O(1/ε) to attain an error of ε under a variant of restricted eigenvalue condition, and the rate has better dependency on the problem dimension than existing methods. Particularly, if the smallest magnitude of the non-zero components of the optimal solution is not too small, the rate of our proposed Hybrid algorithm can be boosted to near the minimax optimal sample complexity of full information algorithms. The core ideas are (i) efficient construction of an unbiased gradient estimator by the iterative usage of the hard thresholding operator for configuring an exploration algorithm; and (ii) an adaptive combination of the exploration and an exploitation algorithms for quickly identifying the support of the optimum and efficiently searching the optimal parameter in its support. Experimental results are presented to validate our theoretical findings and the superiority of our proposed methods.
This looks an interesting result.  Authors suggest a couple of methods, inspired by "mini batch strategy",  for solving linear regression when the signal of interest is sparse and one can observe the data partially.  The problem and their proposed methods are clearly described in the paper.   Theoretical guarantees for convergence of the algorithms are provided which is supported by practical evidence given in the paper.  As the authors suggest, these methods look to outperform other existing methods in term of "sample complexity".  I think result presentation can be improved by paying attention to the following points :  1) I think hiding effect of noise parameter $ \sigma^2 $ in the result presented in table 1 is not a good idea at all. It shows the robustness of the algorithm and omitting it may cause some confusions.  For instance, in the presented result in table 1, it seems just with scaling the problem (sending $ r_min $ to infinity), we can improve the performance of the proposed Hybrid algorithm and make sample complexity independent of $ \epsilon $.  It is not the case because scaling blows up the noise level which is not visible in the way that they present the complexity comparison.   2) It seems choice of the parameter $ s $, defined in line 116, plays a crucial role in the performance of the algorithm.  However, it is not clear to me how the authors choose this parameter for the practical results.  Especially, because the level of sparsity $ s* $ is not known and I assume $ s >= s* $.  It is reasonable to assume an upper bound for $ s* $ and tune $ s $ with standard methods, but neither are mentioned in the paper.  3) I think the comparison in the simulation part is not fair because of the following reasons,   i)  There is no run-time data for the numerical experiments.  Sample complexity is not the only factor that shows the complexity of the algorithm.  In many problems, including compressive linear regression, it is possible to solve the problem with information theoretic bounds for sample complexity at the cost of exponential run-time.  If the claim is having efficient sample complexity, some computational complexity should be included as well.  Moreover, in comparison of different methods, this should be considered.  ii) I think that Dantzig is performing horribly in the synthetic data but pretty well in the real data can be a sign that synthetic data is chosen to artificially fit the proposed method.  4)  The maximum chance that the analysis can guarantee for the success of the Hybrid algorithm, as Theorem 4.4 suggests, is less than 67%, regardless of the number of samples we have.  Although the main point of the work is solving the problem with partially finite observations, not having asymptotic consistency is a weakness for the theoretical result.  In general, I think this is an interesting result while there is a large room for improvement.