Probabilistic Low-Rank Matrix Completion with Adaptive Spectral Regularization Algorithms
We propose a novel class of algorithms for low rank matrix completion. Our approach builds on novel penalty functions on the singular values of the low rank matrix. By exploiting a mixture model representation of this penalty, we show that a suitably chosen set of latent variables enables to derive an ExpectationMaximization algorithm to obtain a Maximum A Posteriori estimate of the completed low rank matrix. The resulting algorithm is an iterative soft-thresholded algorithm which iteratively adapts the shrinkage coefficients associated to the singular values. The algorithm is simple to implement and can scale to large matrices. We provide numerical comparisons between our approach and recent alternatives showing the interest of the proposed approach for low rank matrix completion.
SUMMARY This paper studies the problem of low 
      rank matrix completion which exists in many real-world applications such 
      as collaborative filtering for recommender systems. A previous work (ref 
      [4]) proposed a scalable algorithm called Soft-Impute for solving a convex 
      optimization problem involving the nuclear norm as a regularizer. Like 
      previous work such as probabilistic matrix factorization (PMF), this paper 
      gives the problem a probabilistic interpretation by relating the 
      (non-probabilistic) optimization problem to a MAP estimation problem. 
      Different (concave) penalty functions of the nuclear norm are proposed and 
      then an EM algorithm is proposed to solve the MAP estimation problem. The 
      algorithms proposed in this paper are more general than the Soft-Impute 
      algorithm proposed in [4] in that the latter comes as a particular case. 
      QUALITY The main contribution of this paper is that it 
      proposes new penalty functions based on the nuclear norm for low rank 
      matrix completion. Although experimental validation has shown that the 
      method (HASI) that makes use of the proposed penalty functions generally 
      outperforms other methods compared, its technical soundness could be more 
      strongly articulated via theoretical analysis. This is especially the case 
      when the proposed regularizers are non-convex and hence make the 
      optimization problem more challenging than solving a convex optimization 
      problem. This price would be worth paying only if it had advantages 
      preferably supported well by theoretical analysis, besides experimental 
      validation. Even on the experimental validation, besides showing the 
      results as in Table 1, the reader could gain a deeper understanding of the 
      strengths and weaknesses of different methods if more detailed analysis of 
      the results could also be provided. For example, we only know that HASI 
      usually gives the lowest NMAE except for the MovieLens 1M dataset (which 
      comes second), but we have no clue on what makes the performance degrade 
      in this case. Is it because of the large dataset? Does the proposed method 
      perform less satisfactorily for large datasets due to non-convexity of the 
      optimization problem? Also, there is relatively large variation in the 
      rank among different methods. It would help if the authors could comment 
      on this aspect as well. Since the proposed method is based on 
      point estimation, calling it “Bayesian” may be controversial. It is more 
      appropriate to refer to it as “probabilistic” instead. Related previous 
      work also used this convention. For example, for matrix 
      factorization/completion, a point estimation method is called 
      probabilistic matrix factorization: R. Salakhutdinov and A. Mnih. 
      Probabilistic matrix factorization. NIPS, 2008. while its full 
      Bayesian extension is called Bayesian probabilistic matrix factorization: 
      R. Salakhutdinov and A. Mnih. Bayesian probabilistic matrix 
      factorization using Markov chain Monte Carlo. ICML, 2008. 
      Nevertheless, this is a well-written paper which is a pleasure to 
      read. I also like the supplementary material which includes discussions on 
      generalization to other mixing distributions as well as the binary case 
      which does occur in practice. CLARITY The paper is quite 
      clearly written, making it easy for the reader to follow. I particularly 
      like the organization which presents the complete matrix case first and 
      then considers the more general, incomplete matrix case for matrix 
      completion which is the focus of this paper. Nevertheless, I have 
      a few minor comments. Addressing them will further improve the clarity of 
      the paper. Line 37: “... the low rank assumption is sensitive 
      ...” Do you actually mean “sensible”? I don’t understand why it is 
      “sensitive”. In Eq (7), strictly speaking some constant terms are 
      missing even though they have no effect on the penalty function for 
      optimization. In Figure 3, the labels on the axes are too small to 
      be visible without zooming in the PDF file. Line 184: When you 
      refer to \gamma as regularization parameters, it would help to point to 
      the specific regularized objective function in which they play the role of 
      regularization parameters. It may not be clear to the reader. In 
      Eq (13), it should be the sum of two terms, not their difference. 
      ORIGINALITY Needless to say, matrix completion is not a 
      new problem. It has already aroused a lot of research interests over the 
      past decade. Also, quite a few models have been given a probabilistic 
      interpretation. So, this paper is not novel in this aspect either. The 
      main contribution of this paper is on the introduction of some penalty 
      functions based on the nuclear norm. As far as I know they have not been 
      introduced in the context of matrix factorization/completion and related 
      problems. In fact, many of the techniques used in the paper are 
      existing methods and results too. For example, the global optimality of 
      the M-step of the EM algorithm in Eq. (9) is guaranteed by previous 
      results reported in [11, 12], not this work; the scalability is due to the 
      PROPACK algorithm which is from previous work [4]. If one has to comment 
      on the novelty of this paper, it would be on the combination of these 
      techniques and findings from previous work in a coherent way. 
      SIGNIFICANCE The experimental results reported seem quite 
      good, but it is premature to draw conclusions on the practical 
      significance of the proposed method based solely on these relatively 
      small-scale experiments. Its significance could be articulated more 
      convincingly by making extensions at least along the following directions: 
      - Using larger datasets, e.g., Netflix dataset - Empirical 
      comparison with more state-of-the-art methods, e.g., Bayesian methods, 
      nonlinear (kernel) methods, deep learning methods - Theoretical 
      analysis of new penalty functions