Revisiting Decomposable Submodular Function Minimization with Incidence Relations
We introduce a new approach to decomposable submodular function minimization (DSFM) that exploits incidence relations. Incidence relations describe which variables effectively influence the component functions, and when properly utilized, they allow for improving the convergence rates of DSFM solvers. Our main results include the precise parametrization of the DSFM problem based on incidence relations, the development of new scalable alternative projections and parallel coordinate descent methods and an accompanying rigorous analysis of their convergence rates.
The authors consider the problem of submodular minimization under factorization and incidence assumptions. Specifically, they assume that the function decomposes into terms that depend only on a subset of the variables. They consider the minimum norm problem and two algorithms - alternating minimization (AP) and coordinate descent (CD). In both cases, they improve the old bounds in a pretty natural way - they replace NR by the sum of the neighbourhoods of the components. AP under incidence constraints has been also studied by  Scalable Variational Inference in Log-supermodular Models , Josip Djolonga, Andreas Krause, ICML 2015,  who also extend the results from Nishihara et al, and arrive at exactly the same problem as in this paper (they prove only under the assumption of a regular graph though). The fact that one scales by the inverse of the number of components (i.e. the skewed norms) is not surprising, as similar problems also appear in classical dual decomposition. Their randomized CD algorithm is novel, as they showed that one can not easily extend the results from Ene et al. I find that the intuition behind their approach is not well explained and the discussion could certainly be improved.  I am not sure about the applicability of these models though. For example, in the experimental section they split into 1065 components, but this is necessarily as only two can suffice -- all vertical and all horizontal edges. Similarly, the superpixels typically do not overlap -- if one does several layers they will overlap, but this is typically on the order of 2 to 3. I think if they implement it that way, there won't be any benefits when compared to the classical AP and RCD methods. Moreover, they only implement AP, while Jegelka et al. show that a reflection based approaches obtains much faster convergence. Finally, they measure certified duality gap vs # of iterations - I think the x-axis should be wall-clock time, as the comparison is unfair because different methods do varying number of projections per iteration.  Questions ---  * In the experiments for the superpixel potentials you do not have to use divide and conquer, as a single sort + isotonic regression would suffice, as explained in  Lim, Cong Han, and Stephen J. Wright. "Efficient Bregman projections onto the permutahedron and related polytopes." AISTATS 2016.  * l73, 74 - Maybe note that { i | x_i > 0 } and { i | x_i >= 0 } are both in the lattice of minimizers.  * l121. Can't you say that I(w) consists of R copies of w? It can be a bit confusing in the way it is stated, especially with the variable r appearing twice.  * l234. Not clear why convergence is guaranteed. Aren't you simply showing that you minimize an upper bound? More discussion could certainly help in this section.  * Def 3.8 Maybe say "We say that P is an alpha-proper for some \alpha\in(0,1) if ....". A better name might be alpha-sparse?  * Example 3.1. - Perhaps note that this is exactly a cut function on a chain graph?  * Spell out maybe in more detail why the inequality l* < ... holds? What is the y^* from Def 3.6 for this y?  * l230: unclear if you mean h_{r,i} is an indicator vector on coordinate {r, i} or if for all r\in C only y_{r,i} is nonzero, or i h_{r,*} for r\in C are the only non-zero blocks.  * You can cheaply improve the certificates using isotonic regression, as explained in Bach, [p. 281 ,"Primal candidates from dual candidates."].   Post rebuttal --- In Djolonga et al., they analyze the exactly same problem, even though the motivation is through L-Field. They specifically show that the problems are equivalent and use the same weighted projections, despite proving only the regular case. The authors did not respond to how the results would look like if the natural decomposition is done for the images. I expect their improvements to significantly diminish for this setting. I think the paper is interesting and the setting they consider is natural, although I do not think the practical benefits are well supported by the experiments, as the instances chosen have very natural and simple decompositions.