Polar Operators for Structured Sparse Estimation
Structured sparse estimation has become an important technique in many areas of data analysis. Unfortunately, these estimators normally create computational difficulties that entail sophisticated algorithms. Our first contribution is to uncover a rich class of structured sparse regularizers whose polar operator can be evaluated efficiently. With such an operator, a simple conditional gradient method can then be developed that, when combined with smoothing and local optimization, significantly reduces training time vs. the state of the art. We also demonstrate a new reduction of polar to proximal maps that enables more efficient latent fused lasso.
-- first-round comments -- This paper proposes 
      new methods to compute the polar operator for a class of regularizers 
      based on marginalized modular functions such as group sparsity. 
      Pros: - well-written, and to-the-point. Logically organized 
      - subject is topical (group sparsity, sparsity on DAGS) - Polar 
      operators have not been studied too much (with respect to practical use) 
      compared to proximal operators. The proposed method seems to be quite good 
      - Recent interest in conditional gradient/Frank Wolfe means that polar 
      operators are of interest - Nice secondary results (e.g., prox of lp + 
      TV) Cons: - Prop. 2 allows one to compute the polar via the 
      prox, due to the link between the dual functions and polar functions (and 
      between a function and its conjugate, via Moreau's Thm, eq (43)). By 
      itself, the proposition is not too surprising, and I strongly suspect that 
      a "converse" can also be shown (i.e., one can compute the prox via the 
      polar). If this is true, then does the advantage of the GCG over APG still 
      hold? In the experiments, the proxes used in APG were very slow, but 
      couldn't these be sped up to within a factor of 5 of the time to compute 
      the polar? (since going the other way, it took about 5 evaluations of the 
      prox to compute the polar). Also, it would be nice to say something more 
      about the 1D line search; e.g., can one use Newton's method, as in SPGL1 
      (Friedlander and van den Berg)? - Actually, is it not possible to 
      find the polar via the prox in another manner? The polar operator for a 
      set S at the point "d" is polar(d)=argmin_{x in S} -d^T x. The prox 
      operator at the point "y" is prox(y)=argmin_{x in S} .5||x-y||^2. Then by 
      expanding the quadratic, we roughly get lim_{alpha -> infinity} prox( 
      -alpha d ) = polar(d). Computationally there are still issues, but it may 
      be an interesting alternative. - Regarding the exact solution of 
      1D TV, improving on that in [25]. I do not think this is new; please check 
      recent literature like L. Condat, "A direct algorithm for 1D total 
      variation denoising," preprint hal-00675043, 2011 (which also mentions 
      other exact, direct algorithms, such as the taut string algorithm). 
      - complexity results used Nesterov's smoothing method, but 
      experiments use L-BFGS. Then, line 362, the paper says "...the method 
      proposed in this paper is not only faster... in practice, it also 
      possesses a strictly better worst case time complexity..." The practical 
      results use L-BFGS, and the worst case time complexity use Nesterov's APG, 
      so this statement is not correct unless you have a complexity bound for 
      L-BFGS. - The appendix is rather long and contains new 
      propositions. I prefer appendices to be reserved for small details and 
      lemma. If such a long appendix is necessary, is NIPS really the right 
      venue? Neither a "pro" nor a "con", this paper will likely be hard 
      to read for most readers, since it combines combinatorial-like functions 
      (modularity), convex optimization (polars and duals), and 
      statistics/machine learning (lasso, etc.) In my opinion, the pros 
      outweigh the cons considerably. I think this is a significant paper. 
      ------------------------ new comments after reading feedback: 
      - In response to the other reviewer, you wrote "APG also needs to 
      evaluate the loss function if the Lipschitz constant requires adaptive 
      estimation". Actually, it is possible to do a check using just the 
      iterates and the gradients since if the gradient is Lipschitz it is also 
      firmly non-expansive (when scaled by the Lipschitz constant), and so you 
      can use the firmly non-expansive bound to check (see the old 
      Eckstein-Bertsekas paper on the Douglas-Rachford iteration). Also, 
      this comment "- complexity results used Nesterov's smoothing 
      method..." was not addressed. You are claiming to be better in theory 
      and practice, but your theory uses APG and your practice uses L-BFGS, so 
      theory and practice do not mach.