A Bridging Framework for Model Optimization and Deep Propagation
Optimizing task-related mathematical model is one of the most fundamental methodologies in statistic and learning areas. However, generally designed schematic iterations may hard to investigate complex data distributions in real-world applications. Recently, training deep propagations (i.e., networks) has gained promising performance in some particular tasks. Unfortunately, existing networks are often built in heuristic manners, thus lack of principled interpretations and solid theoretical supports. In this work, we provide a new paradigm, named Propagation and Optimization based Deep Model (PODM), to bridge the gaps between these different mechanisms (i.e., model optimization and deep propagation). On the one hand, we utilize PODM as a deeply trained solver for model optimization. Different from these existing network based iterations, which often lack theoretical investigations, we provide strict convergence analysis for PODM in the challenging nonconvex and nonsmooth scenarios. On the other hand, by relaxing the model constraints and performing end-to-end training, we also develop a PODM based strategy to integrate domain knowledge (formulated as models) and real data distributions (learned by networks), resulting in a generic ensemble framework for challenging real-world applications. Extensive experiments verify our theoretical results and demonstrate the superiority of PODM against these state-of-the-art approaches.
Paper summary:    The paper proposed a learning based hybrid proximal gradient method for composite minimization problems. The iteration is divided into two modules: the learning module does data fidelity minimization with certain network-based priors; consequently the optimization module generates strict convergence propagations by applying proximal gradient feedback on the output of the learning module. The generated iterates were shown to be a Cauchy sequence converging to the critical points of the original objective. The method was applied to image restoration tasks with performance evaluated.      Comments:      The core idea is to develop a learning based optimization module to incorporate domain knowledge into conventional proximal gradient descent procedure. One issue with such a hybrid scheme is that by definition the output of the learning module is acceptable to the proximal gradient module only when the bounding condition in Eq.(9) is met. This then raises a serious concern that the learning module could have very limited effect on the iteration procedure if its output is frequently rejected by the proximal gradient module, i.e., the bound (9) is frequently violated. If that is the case, then the proposed method is no more than a proximal gradient method. So it will be helpful to provide more insightful explanations on the condition (9), e.g., to discuss how hard is it to be fulfilled.    Another main concern is the readability of paper. The model description part in Section 3 is rather abstract and hard to follow in general. To gain better intuition of method, the reviewer suggests providing one illustrating example or two such as those appeared in Section 5 to aid model elaboration.   The convergence analysis falls on the weak side: the procedure was only shown to converge asymptotically to critical points, using fairly standard proof skills in proximal gradient analysis. The rate of convergence remains largely unknown, even for convex cases.   Strong points:  + The attempt to bridge model optimization and deep propagation is somewhat interesting and novel.  + Numerical study is extensive.  Weak points:   - The motivation and significance of this work are not fully justified in text and mathematical analysis.  - Some important technical details about the proposed algorithms are missing (see the comments above). - The writing quality needs to be significantly improved.   === Updated review after author response ===  I further checked the technical proofs in more details and spot a seemingly fatal flaw in the proof of Proposition 1 (Line 46 - 47 of the supplement): i$x_G^l$ is claimed as a global minimizer of (10). However, as far as I can tell, $x_G^l$ is no more than a first-order stable point of (10). Since the primal interest of this work is on nonconvex problems, it is obvious false to claim that a stable point is a global minimizer.   In the meanwhile, after reading the author response, I found most of my major concerns were not clarified in a satisfying way. Particularly, the following issues are still there:  1. The problem statement is quite confusing: the original optimization problem is defined in Eq. (1) but the algorithms are designed for solving (2). The connection between these two formulations is not well explained. In particular, the inner product error term in Eq. (2) does not make sense to me.   2. Unless the authors can show that the bounding condition (9) is eventually always true in iteration, the proposed algorithm essentially reduces to a standard PG descent procedure.   3. The convergence analysis turns out to be weak and of limited novelty: it mimics largely the conventional PG analysis without introducing any conceptually new proof techniques and/or results.   All in all, I still have serious concerns about this submission and vote strongly for reject.