Online Optimization for Max-Norm Regularization
Max-norm regularizer has been extensively studied in the last decade as it promotes an effective low rank estimation of the underlying data. However, maxnorm regularized problems are typically formulated and solved in a batch manner, which prevents it from processing big data due to possible memory bottleneck. In this paper, we propose an online algorithm for solving max-norm regularized problems that is scalable to large problems. Particularly, we consider the matrix decomposition problem as an example, although our analysis can also be applied in other problems such as matrix completion. The key technique in our algorithm is to reformulate the max-norm into a matrix factorization form, consisting of a basis component and a coefficients one. In this way, we can solve the optimal basis and coefficients alternatively. We prove that the basis produced by our algorithm converges to a stationary point asymptotically. Experiments demonstrate encouraging results for the effectiveness and robustness of our algorithm.
The paper considers a problem where a matrix Z is to be decomposed into a low-max-norm matrix X, and a sparse matrix E, which is formulated as a convex problem with max-norm and L1 regularization. This can be solved using batch methods, but the focus here is on an online algorithm, which does a single pass over Z's rows. The main trick here is to reduce the original problem to another problem, which decomposes over Z's rows, and hence can be attacked using an online learning procedure. The paper proposes an algorithm, proves that it asymptotically converges to a stationary point, and provides experiments on synthetic data.I liked the basic idea of considering online matrix decomposition using the max-norm, which is an interesting and less-explored alternative to the nuclear norm. The algorithm derivation uses a simple but non-trivial trick, and the experiments seem promising. On the other hand:- I had problems with understanding the motivation to the problem through matrix completion: First, in matrix completion the data matrices are very sparse, so there is no reason not to use batch methods. Second, I couldn't follow the explanation in section 1.2 why matrix completion can be cast into this framework: I understand that the equation in line (80) is equivalent to matrix completion when C -> infinity, but why is it true that the equation is equivalent to the proposed framework (equation 1.1)? I'd appreciate it if the authors can explain this in their rebuttal.- The algorithmic derivation appears to be quite similar to OR-PCA (cited paper [8]), with the twist of using the max-norm rather than the nuclear norm, and an additional trick to get decomposability along rows. This makes this work a bit incremental.- The analysis, although far from trivial, results in a rather weak asymptotic convergence guarantee to some stationary point. This appears to be weaker than what can be obtained for related algorithms which use other approaches, such as OR-PCA. - The algorithm is computationally demanding: At each iteration, we need to solve a d*d matrix optimization problem, where d >= ambient data dimension to the best of my understanding (see detailed comments below). Although warm-start partially alleviates this, it's not clear if we end up with an algorithm whose runtime is much better than a batch method. Overall, I think the paper has some interesting ideas, but also a few significant weaknesses.Other issues/comments--------------------- Assumptions in section 4.1: It is stated that they can be made valid by adding a bit of regularization, but this also introduces additional terms into the optimization problem, so it's not clear if the result would still hold.. This should be clarified.- Is it possible to use the algorithm in a batch setting, by sub-sampling individual rows uniformly at random?- In the proposed algorithm, one needs to point out how d should be chosen. To the best of my knowledge, one needs to pick d >= max{p,n} to ensure that proposition 2.1 still holds. In that case, this issue should be explicitly discussed.Minor comments------------------- The manuscript is in A4 size rather than letter size.- line 45: "even" -> "even if"- line 56: "...E, we aim..." - comma should be period.- The introduction motivates the max-norm but formulation (1.1) also includes an additional E component with L1 regularization - this should be more clearly motivated (e.g. to handle sparse contamination)- Line 88: "in literature" -> "in the literature".- Line 173: For equation 2.7 to be true, one needs to assume that each row of the matrix Z is sampled i.i.d. from some underlying distribution. I didn't see this explicitly stated anywhere..- Line 181: "alternative" -> "alternating"(?)- LIne 232: Should ||r||_2 be ||r||_2^2? Otherwise the expression may not be strongly convex.- Line 350: "groundtruth"- Experiments: What value of d did you take? - Line 404: "In specific" -> "specifically"- The authors mention at several places that the max-norm is a tighter approximation of the rank than the nuclear norm. To the best of my knowledge, this is imprecise: It's a different relaxation, using a different underlying norm assumption on the latent factors (see for instance the introduction in the cited paper [13]). At most, one can argue that it appears to be a better approximation in many cases. - It's hard to see from Figure 1 that OMRMD is better than OR-PCA: They both look more or less the same.