An Approximate, Efficient Solver for LP Rounding
Many problems in machine learning can be solved by rounding the solution of an appropriate linear program (LP). This paper shows that we can recover solutions of comparable quality by rounding an approximate LP solution instead of the exact one. These approximate LP solutions can be computed efficiently by applying a parallel stochastic-coordinate-descent method to a quadratic-penalty formulation of the LP. We derive worst-case runtime and solution quality guarantees of this scheme using novel perturbation and convergence analysis. Our experiments demonstrate that on such combinatorial problems as vertex cover, independent set and multiway-cut, our approximate rounding scheme is up to an order of magnitude faster than Cplex (a commercial LP solver) while producing solutions of similar quality.
Summary: The paper considers LP relaxations in 
      machine learning and studies the problem of solving the LP relaxation 
      only approximately and derives bounds on the overall approximation 
      rate when solving the integer program (by rounding the LP solution). 
      Pros: - very interesting problem (solve LP relaxation, knowing 
      that in the end we will round the solution). This is a relevant 
      problem in many machine learning fields. - to the best of my 
      knowledge the first work that gets approximation guarantees for the 
      overall algorithm (LP approximation by smoothing, only solve smoothed 
      LP to a some threshold, integer rounding). Cons: - I would 
      skip the SCD algorithm in the paper and rather try to give more 
      intuition and proof sketches in the main part of the paper. I do not 
      think a reader can appreciate section 3.3 too much. It looks too me 
      that the running time is not very exciting, both n and 1/epsilon^2 
      appear (SGD does for example not have the dependence on n in SVMs), 
      why not solve the problem with a more standard solver? It seems to me 
      SCD is not the main contribution of the paper. - the speed 
      improvements seen in the experiments are not hugely impressive 
      (roughly one order of magnitude). Also there is probably some 
      precision option in CPLEX, what happens if you lower the precision in 
      CPLEX LP? - few language problems, nothing major. - long appendix 
      Quality: Experiments could be improved, some questions about 
      the main theorems remain. Clarity: Language problems and 
      suboptimal choices in the presentation (e.g. get rid of 3.3 and 
      include more material from the appendix). Originality: To the 
      best of my knowledge the main section (3.2) is original. 
      Significance: LP relaxations are important tools for 
      combinatorial optimization in machine learning. Comments: 
      39: Despite ? 40: .. Gurobi cannot scale .., the order 10 of 
      speedup of your method is probably not going to change this. 43: 
      -- but often much more efficiently. Rewrite. 52: Cplex do no scale to 
      52: The Cplex LP solver was slower but it ran through in the time 
      limit on almost all problems. 55: Obviously message passing is 
      related in that it also solves LP relaxations, but I fail to see the 
      connection to your method. The smoothing? 69: in which to articulate 
      74: would appreciate some citations 98: is a subset of V each edge 
      (rephrase!) 102: vertex cover as a 104: for each vertex v \forall 
      v \in V, rephrase! 130: one can easily 132: and an example such 
      scheme (rephrase) 144: Figure caption needs more details about the 
      parameters. (3): min_x (4): max_u 167: is the definition 
      standard? if so, cite 177: \prod_[0,1]^n was not really introduced 
      176: to the linear program in 193: how is d defined, it is a 
      concatenation of vectors and a matrix, what's the norm on d? 191: 
      the paragraph here seems to apply to general LPs, it is a bit confusing, 
      as it's introduced after (5) 236: \beta depends on optimal 
      solution, how would we estimate this? 280: algorithm 2 is described 
      (rephrase) 304: what does thetis stand for? 310: have rarely been 
      used? for 315: Boolean 317: the resulting graphs 320: Gibbs 
      sampling is not really state of the art. How exactly, annealing? 321: 
      why didn't you also use an LP MAP solver such as the one in [14]? 331: 
      have placed if (rephrase) Figure 2: what about energies (apart of P, 
      R, F1)? 357: in this paragraph you mention the appendix twice 360: 
      the second family of 369: this experiment is to compare 418: 
      Cplex-IP on large scale 960: describe difference between Table 3 + 4 
      in more detail