Robust Regression via Hard Thresholding
We study the problem of Robust Least Squares Regression (RLSR) where several response variables can be adversarially corrupted. More specifically, for a data matrix X ∈ R p×n and an underlying model w * , the response vector is generated as y = X T w * + b where b ∈ R n is the corruption vector supported over at most C · n coordinates. Existing exact recovery results for RLSR focus solely on L 1 -penalty based convex formulations and impose relatively strict model assumptions such as requiring the corruptions b to be selected independently of X. In this work, we study a simple hard-thresholding algorithm called TORRENT which, under mild conditions on X, can recover w * exactly even if b corrupts the response variables in an adversarial manner, i.e. both the support and entries of b are selected adversarially after observing X and w * . Our results hold under deterministic assumptions which are satisfied if X is sampled from any sub-Gaussian distribution. Finally unlike existing results that apply only to a fixed w * , generated independently of X, our results are universal and hold for any w * ∈ R p . Next, we propose gradient descent-based extensions of TORRENT that can scale efficiently to large scale problems, such as high dimensional sparse recovery. and prove similar recovery guarantees for these extensions. Empirically we find TOR-RENT, and more so its extensions, offering significantly faster recovery than the state-of-the-art L 1 solvers. For instance, even on moderate-sized datasets (with p = 50K) with around 40% corrupted responses, a variant of our proposed method called TORRENT-HYB is more than 20× faster than the best L 1 solver.
"If among these errors are some which appear too large to be admissible, then those equations which produced these errors will be rejected, as coming from too faulty experiments, and the unknowns will be determined by means of the other equations, which will then give much smaller errors."
A. M. Legendre, On the Method of Least Squares. 1805.
This paper proposes and analysis a new algorithm for the robust regression problem. Robust regression problem is an old problem, related to outliers detection.Introduction ---------------The robust regression problem is defined in Eq. (2). I think that the authors should slightly rephrase their definition as Eq. (2) is one possible "modern" definition of the robust least square problem. Authors also claims that robust regression and sparse regression are "completely different". The only difference if the sparse assumption on the coefficients vector w. The robust regression has a much more specific model on the noise, but can also be a sparse regression (as studied in Sec. 5). Again, I think that the authors should rephrase this part to avoid miss-understanding.My major remark here, is a lake of state of the art on robust regression/outliers detection. Indeed, only very recent work are mentioned, but this a very old problem, and some classical methods such as L1 regression (in the sense of min || y - X w ||_1 ) should be mentioned here, and used as a reference.Sec 2 ------The problem formulation is clear, and the noise is specified thanks to two terms : - a classic gaussian noise - an unbounded "sparse" noise.However, this formulation is not the "only one" possible. Maybe one reference on some Bayesian model for the noise on robust regression would be welcome here, in order to have a quite complete point of view on robust regression.Moreover, I think that the authors could reformulate clearly the robust least square regression problem as an optimisation problem such as (there is some others possibilities) min_{w,b} 0.5 ||y - Xw - b||^2 + \lambda ||b||_0Sec. 3 -------This section introduces the "Torrent" algorithms studied in Sec. 4. This algorithms made me think of a kind of "matching pursuit": the selection of the active set is made directly on the residual instead on its correlation with the matrix X. Torrent-FC being the "OMP" and Torrent-GD the "classical" MP.The definition of Hard thresholding suits well with the problem, but is a kind of counter-intuitive compared to the classical hard thresholding used for sparse regression.  Sec. 4 ------This section presents the theoretical results of the proposed algorithm, and then is the main theoretical contribution of the paper. I have no particular remark on this part.Sec. 5 -------This section extends previous results to the "Robust sparse regression problem". Following my remark on Sec. 2,I think that at this point, the author should re-formulate the problem asmin_{w,b} 0.5 ||y - Xw - b||^2 + \lambda ||b||_0 + \mu ||w||_0 Sec. 6 -------The experimental part if ok, but the only method the authors use for comparison is L1-DALM which solve only the sparse robust regression problem, not the robust regression problem without the sparse assumption. So the comparison is not fair, as the goal of the two approached is not the same. I also think that the author must compared their algorithm to a classic L1-regression problem, which is the state of the art of robust regression (and also a L1-L1 approach for the robust sparse regression problem).