Efficient Algorithms for Non-convex Isotonic Regression through Submodular Optimization
We consider the minimization of submodular functions subject to ordering constraints. We show that this potentially non-convex optimization problem can be cast as a convex optimization problem on a space of uni-dimensional measures, with ordering constraints corresponding to first-order stochastic dominance. We propose new discretization schemes that lead to simple and efficient algorithms based on zero-th, first, or higher order oracles; these algorithms also lead to improvements without isotonic constraints. Finally, our experiments show that non-convex loss functions can be much more robust to outliers for isotonic regression, while still being solvable in polynomial time.
This paper considers the problem of continuous submodular minimization with box constraints and extra isotonic constraints, motivated by nonconvex isotonic regression. The paper gives an elegant algorithm for solving these constrained problems, by leveraging isotonic structure that is already present in the algorithms for continuous submodular minimization with box constraints from [1]. Beyond generalizing the approach of [1] to these constrained problems, the paper further shows that an extra smoothness condition on the objective yields better dependence the suboptimality criterion, for free, than thought before. Higher order smoothness can yield even better dependence. Finally, the paper shows the practical relevance of these techniques via a seemingly simple robust isotonic regression problem where natural convex formulations fail. A different (student t) model succeeds, but the objective is nonconvex; heuristics give poor performance, while the techniques from the paper solve the problem optimally and yield good performance on the regression task.  Overall I think the paper has clean, well-executed theory and a motivating practical problem. Continuous submodular optimization is an important topic relevant to the NIPS community, as it presents a class of tractable nonconvex problems of use in machine learning applications (like nonconvex isotonic regression). Within this area of work, the paper is original and significant: while continuous submodular _maximization_ has seen much activity and many positive results in recent years, continuous submodular _minimization_ (this paper) seems more elusive. (beyond this paper I can only think of Bach '18 [1] and Staib and Jegelka '17, versus a multitude of maximization papers). This paper grants to us a new class of tractable constrained nonconvex problems.  The paper is also clearly written and was a joy to read.  I do have a few minor concerns/questions -- these are all practical considerations as I think the math is clean and not fluffy: - what is the wall-clock runtime like? - is it obvious that majorization-minimization is the most competitive alternative? (e.g. vs running projected gradient descent from random initialization) - I find the practical benefit of extra Taylor expansion terms in the approximation somewhat suspect -- it seems like you have to do a lot of extra work (both algorithmically and just by having to query q different derivative oracles) to get the improvement in epsilon dependence. It seems the fancy discretization was not really even used in the experiments (in "Non-separable problems" the authors compare the quality of approximations, but they seem to just use the usual discretization for the main regression experiments and just note that smooth objectives automatically get better approximation). Thus I am skeptical that "the new discretization algorithms are faster than the naive one." However the observation that smoothness implies better speed for free is still important.  Overall, I think this is a solid paper with significant theoretical and algorithmic contribution, and it is also well-written. Some aspects of the experiments could be strengthened, but still I think this is a clear accept.   minor comments: - the SDP relaxation is based on [15] which should be cited in the main text (it is cited in the appendix) - the point that \hat H is not technically submodular but we can still approximately solve probably should be stated more formally