Quadratic Decomposable Submodular Function Minimization
We introduce a new convex optimization problem, termed quadratic decomposable submodular function minimization. The problem is closely related to decomposable submodular function minimization and arises in many learning on graphs and hypergraphs settings, such as graph-based semi-supervised learning and PageRank. We approach the problem via a new dual strategy and describe an objective that may be optimized via random coordinate descent (RCD) methods and projections onto cones. We also establish the linear convergence rate of the RCD algorithm and develop efficient projection algorithms with provable performance guarantees. Numerical experiments in semi-supervised learning on hypergraphs confirm the efficiency of the proposed algorithm and demonstrate the significant improvements in prediction accuracy with respect to state-of-the-art methods.
In this paper the authors consider the problem of minimizing sums of squares of Lovasz extensions. This is a convex problem and the authors derive its dual and propose several techniques for solving it that have a linear convergence rate. The consider problems over graphs - transductive learning and pagerank - and I am not aware of other problem classes that can be cast into this framework. What I find slightly confusing is why the authors posed the problem in terms of submodularity and Lovasz extensions, when, unless I'm mistaken, their theory and algorithms work for any polyhedral support function, not necessarily over base polyhedra. If I'm mistaken, it would be nice to point out what is the barrier to applying their methods beyond submodular polytopes. Moreover, unlike for submodular functions, there are no results about the quality of the rounded solution if we treat the problem as a continuous relaxation of the discrete problem.  In the dual the authors obtain a dual containing cones derived from the base polytope and they develop algorithms for projecting onto them. They use Frank-Wolfe and a variant of it (Fujishige's min-norm algorithm is essentially a variant of fully corrective Frank-Wolfe). What is interesting is that they use FW which is classically defined only for compact domains and apply it to cones. They keep the same 1/k rate, and it would be quite useful if their analysis can be extended to other families of cones "generated" by polytopes.  I unfortunately cannot comment much about the quality of the obtained solutions in the experimental section as I am not very familiar with the recent literature on transductive learning. I am interested why they did not compare against label propagation, which is a widely used method.  Questions and remarks:  1) The Lovasz extension is not necessarily positive. I guess you assume F to map to [0, oo) instead of R.  2) Perhaps you should point out in line 126 that you are using the conjugate representation of the quadratic.  3) What would happen if you introduce linear terms in the objective? How would your algorithms change?  4) Why do you present the coordinate descent method in section 4, when you obtain the best results using AP?  5) Why not also plot the value of the function you are minimizing in the experimental section, which is more important than the duality gap.  6) It would be interesting to see how does accuracy change as a function of cpu time. I am not sure if having gaps of the order of 1e-10 makes a big difference, a the problems are defined using noisy data.  7) If I'm looking at it right, in Figure 1 it is the case that gaps of 1e-8 and 1e-9 can have a quite different classification performance (3% vs 9% error). Is the top figure not representative of the gaps you obtain for the lower table, or is something else going on?  8) What is the submodular function corresponding to (10)? It is not in your original table due to the presence of sqrt(w_ii) and sqrt(w_jj) in the denominator.  9) How does the accuracy look like for the real data? You're only showing duality gaps, and not comparing the accuracy of the existing work.   ----  The rebuttal did address the questions that I had. To conclude, while I can not evaluate the impact on the transductive learning problems, I think this paper has several interest results on submodular optimization and the Frank-Wolfe algorithm that could be interesting to parts of the NIPS community. Hence, I am leaning towards accepting this submission.