Probabilistic low-rank matrix completion on finite alphabets
The task of reconstructing a matrix given a sample of observed entries is known as the matrix completion problem. It arises in a wide range of problems, including recommender systems, collaborative filtering, dimensionality reduction, image processing, quantum physics or multi-class classification to name a few. Most works have focused on recovering an unknown real-valued low-rank matrix from randomly sub-sampling its entries. Here, we investigate the case where the observations take a finite number of values, corresponding for examples to ratings in recommender systems or labels in multi-class classification. We also consider a general sampling scheme (not necessarily uniform) over the matrix entries. The performance of a nuclear-norm penalized estimator is analyzed theoretically. More precisely, we derive bounds for the Kullback-Leibler divergence between the true and estimated distributions. In practice, we have also proposed an efficient algorithm based on lifted coordinate gradient descent in order to tackle potentially high dimensional settings.
** FROM AUTHOR FEEDBACK AND DISCUSSION WITH OTHER REVIEWERS **From discussion with other reviewers, the bounds in this paper are much better than previous results, but the difference and impact are poorly explained:BOUND AND IMPACT (from discussion)-- The squared Hellinger is dominated by Kullback, hence a bound with Kullback is stronger than Hellinger...-- ...the bounds proposed in this paper are really much better than previous results, but this is indeed badly explained in the paper. The bounds proposed in this paper are of order (assuming square d * d matrix to simplify) max( r d log d / n, r (log d / n) ^(1/2) ) where r = rank, d = number of rows = number of columns and n number of observed entries while previous bounds from [6] and [8] are of order r (d / n) ^ (1/2) Recalling that in matrix completion problem one typically has n >> d (otherwise no recovery is possible), this is a real improvement. Actually the result from this paper matches the results from Koltchinskii Tsybakov and Lounici (2011), that gives the best generalization errors for matrix completion to be found in literature. -- one doesn't need to know an upper bound for the rank, while [6] does.SUMMARY:Quality score updated to 6 due to novel theoretical contribution, after consultation with other reviewer (above).If the practical benchmark wasn't lacking (there are many standard baselines for MovieLens which are ignored), quality would be 7.If the paper discussed and illustrated the practical significance of T2, quality would be 8.------------------------------------------------MAIN IDEAS OF THE PAPERThe paper addresses the reconstruction of a matrix based on partial observations. The entries are discrete. It is assumed that the discrete entries are generated from a likelihood function f(X) that depends on matrix X (for the binary case) or p-1 matrices X_p for p categories. It is done by maximizing the likelihood over the observed entries (the same entry index can be observed more tha once, according to some \Pi distribution), but regularizing the model with a nuclear or trace norm constraint on X.As the true X is unknown, we would like to know how far off X^ is, in terms of the Kullback-Leibler (KL) divergence between f(X) and f(X^). For this two upper bounds are given. A version of the Rank 1 Descent (R1D) algorithm is given to solve for X^.The bounds are for 1-bit (p=2); experimental results are for the general case.The experiments are on a simulated data set, and MovieLens100K. In it, the paper shows that when the data is sampled from a low-rank model with logistic likelihood, the algorithm better recovers the true sampling probabilities than one that assumes a Gaussian likelihood instead of a logistic one. On MovieLens100K, which has ratings 1 to 5, the logistic model has a smaller predictive error than a Gaussian model.QUALITY / CLARITYThe paper seems to stand on two disconnected legs. An upper bound on the KL divergence between f(X) and f(X^) is provided in Theorem 2 for 1-bit matrix completion, but the paper does not explain or show its practical significance. It might be informative if the experiments have bearing on the developed theory, for instance, by including the KL bounds from T2 in Figure 1. Understandably, the bound relies on unknown constants, in which case a relevant interpretation of the bound is required.For these reasons I’m not sure the reader is adequately informed. After the main bound is stated, it is never stated, discussed, or evaluated.ORIGINALITYThe paper addresses a low-rank matrix completion for categorical random variables, which is a new contribution.There is a marginal extension of R1D (Algorithm 1).SIGNIFICANCEIt is not clear that the state of the art is advanced in a demonstrable way. There is a copious amount of literature evaluating 1 to 5 star rating prediction on the MovieLens and Netflix data sets, all of which are evaluated baselines that can be copied verbatim from known work.ADDITIONAL COMMENTS / SUGGESTIONSThe conclusion states that the rates of convergence in T2 are faster than that of Davenport and Cai's. I did not read their papers (Davenport's arxiv.org/abs/1209.3672) and Cai and Zhou (JMLR 2013) in extreme detail, but it seems that their bounds are in terms of Hellinger distance and Frobenius norm? It would be nice to know from the paper how the rates are faster. If there is a rebuttal period, will the authors please add this information? And clearly demonstrate how the bound advances the state of the art, or gives new insights?The binary case has been studied before outside the “1 bit” setting, for instance in [Matchbox: Large Scale Bayesian Recommendations; Stern, Herbrich, Graepel, in WWW 2009]. The likelihoods are similar to that employed in the paper (logit, probit), although the rank is explicitly constrained.Would the likelihood in the baseline evaluation (line 315) be the author's model (replacing p with X), except that a probit instead of logit link function is employed?For discrete ordinal ratings, a stochastic ordering is assumed in the input space, so that for four categories “awesome” > “good” > “mediocre” > “bad” are stochastically ordered [Regression models for ordinal data; McCullagh, in Journal of the Royal Statistical Society B, 1980]. It is a property that the model does not capture, but one that is very applicable to the MovieLens100K evaluation. See for instance [A Hierarchical Model for Ordinal Matrix Factorization; Paquet, Thomson, Winther, in Statistics and Computing, 2012] for ordered discrete alphabet and movie ratings.The paper could demarcate where / how R1D is extended.Lines 192 and 358: T2 suggests a setting for lambda.Line 410: Offsetting or shifting with a bias is common in the “Gaussian model” in literature: why not mean-centre the ratings? Analogously, it’s done for Principal Component Analysis.Typos, etc.Line 90: Shatten q-norm defined, never used (except q=1).Line 98: Hellinger distance defined, never used.Sections 3.1 and 3.2. Notation between e^j and h^j is switched around Line 220?Lines 219 and 250: Multiple definitionsLine 355: Figure 3.3Line 360: Table 3.3 and 3.3