Sparse Support Recovery with Non-smooth Loss Functions
In this paper, we study the support recovery guarantees of underdetermined sparse regression using the 1 -norm as a regularizer and a non-smooth loss function for data fidelity. More precisely, we focus in detail on the cases of 1 and ∞ losses, and contrast them with the usual 2 loss. While these losses are routinely used to account for either sparse ( 1 loss) or uniform ( ∞ loss) noise models, a theoretical analysis of their performance is still lacking. In this article, we extend the existing theory from the smooth 2 case to these non-smooth cases. We derive a sharp condition which ensures that the support of the vector to recover is stable to small additive noise in the observations, as long as the loss constraint size is tuned proportionally to the noise level. A distinctive feature of our theory is that it also explains what happens when the support is unstable. While the support is not stable anymore, we identify an "extended support" and show that this extended support is stable to small additive noise. To exemplify the usefulness of our theory, we give a detailed numerical analysis of the support stability/instability of compressed sensing recovery with these different losses. This highlights different parameter regimes, ranging from total support stability to progressively increasing support instability.
The authors present sharp criteria for l_1-sparse support recovery under small noise in the context of non-smooth regression losses l_\infty and l_1. Traditionally, the problem studied in support recovery left the loss shape fixed at MSE (l2^2) loss and explored different non-smooth penalties, starting from the work of Fuchs on l1 penalty up until the recent, possibly most general partly smooth penalties of Vaiter et al. In the present contribution, the authors keep the simple l1 penalty but explore the effect of changing the loss function. Concretely, two loss functions, l_\infty and l_1 are studied. Both share the characteristic of being polyhedral.  A proof for both cases is given, along with some confirmatory experimental results.
This paper presents a new angle of generalization of the l_1 support recovery criteria to other types of loss functions, which are generally useful in regression: l_\infty corresponds to a uniform noise assumption, whereas l_1 corresponds to a sparse noise assumption. In performing this generalization, it exhibits the functionality of some quantities also obtained in the classic MSE + l1 setting, but which at the time were too specific to be given names. We thus have a non-trivial step forward in the understanding of the mechanisms involved. The proofs in the main paper are sound.  However, my main comments concern the readability of this contribution. This paper took me by far the longest to review, because while mathematically perfectly rigorous, it is as if the authors a) underestimate the fact that their contribution is some non-trivial convex analysis, that they b) do not seem to be interested in making any concessions to the reader by providing their own intuitions or figures elucidating them and c) that this may lead to reviews that completely miss the point of the paper.  Specifically, my points are  * It would be great to have a small diagram explaining the regression problem in 1D or 2D, especially since l_1 and l_\infty losses are equal, resp. equivalent in these two settings. There is sufficient space to add this without deleting anything.  * up until a certain point in the paper, the paper works with general dual pairs \alpha, \beta whose harmonic mean is 1. At some point this specializes to the polyhedral losses and later in the paper specializes to the l_\infy loss (l1 in supp mat). These transitions have to be made more clear. It would be helpful also to state what intuitions are needed to keep the general (\alpha, \beta) (i.e. \alpha \in (1, \infty) is not polyhedral so the proof is different, but here is how you would go about it, but here is why we don't do this in this paper).  * In general, the first reading of this paper is difficult, because many notations are introduced a posteriori. One arrives at table 1 and spends some time looking for definitions of \tilde\Phi and S. These are introduced in the lines below. Then one assumes |S| = |J|, which seems surprising, and is referred forward to Lemma 2 for discussion, instead of giving a quick intuition that this is not only "for simplicity" but even "almost surely holds" in a probabilistic setting and is a reasonable assumption, before referring to the details mentioned in lemma 2. There are also several formulae which introduce a symbol and state in the next lines ", where [symbol] means [meaning]". Could these be put before the formula?  * l120 concerning the situation where I != J, there is also [Tibshirani, The Lasso problem and uniqueness]. While the paper seems to state something that is implicit in many previous contributions, it would be worth being a little more explicit in how these relate  * It took me 3 readings to see the bold r in regularization in l139 to make the link to FO_r. Please introduce explicitly FO_r and FO_\beta. Additionally, it would be helpful to make it a bit clearer, by adding an intermediate step, what "corresponds" means in l139. Not all readers are versed enough in convex duality to read this easily. In general, since there still remains half a page of space, a way of obtaining the Fenchel dual (e.g. by introducing the lagrangian, splitting primal variables and minimizing it wrt both of them) and first order conditions could also be given.  * It would be great to have a bit more intuition on the Lagrange multipliers v_\beta (and some more intuition on p_\beta can be helpful too). In which space do they live (the same as x resp. same as y). How can one imagine them geometrically? The polytope defined by the regression constraint is difficult to untangle in the space of x, since one doesn't know which constraints are active. This polytope is an intersection of seminorm balls, which could even be illustrated in 2D. Diagrams, although not easy to make, would help readers enormously.  * between line 189 and 190, does it have to be p_{1, S}?   Minor * l192 "So now, " -> "Now " * l194 hypotheses -> hypothesis * between l200 and l201 put a word between the formulas "or the second line which implies the first by operator inequality" *l202 get -> obtain *l207 get -> obtain *l248 exceeds -> exceed *l255 theoertical -> theoretical