Binary Rating Estimation with Graph Side Information
Rich experimental evidences show that one can better estimate users' unknown ratings with the aid of graph side information such as social graphs. However, the gain is not theoretically quantified. In this work, we study the binary rating estimation problem to understand the fundamental value of graph side information. Considering a simple correlation model between a rating matrix and a graph, we characterize the sharp threshold on the number of observed entries required to recover the rating matrix (called the optimal sample complexity) as a function of the quality of graph side information (to be detailed). To the best of our knowledge, we are the first to reveal how much the graph side information reduces sample complexity. Further, we propose a computationally efficient algorithm that achieves the limit. Our experimental results demonstrate that the algorithm performs well even with real-world graphs.
Summary of paper: The paper considers the problem of completing a binary rating matrix, and theoretically studies whether side information in terms of graphs among users or items aid in the matrix completion task. On a technical level, the study assumes that the users belong to two communities, and users in the same group rate all items identically (upto some noise). The community structure is further revealed by the side information in terms of a graph generated from stochastic block model. The paper derives the optimal sample complexity for rating estimation in terms of the expected number of ratings that needs to be seen to exactly recover the rating matrix.  Review summary: The paper is a solid piece of theoretical work and clearly written. The problem tackled is quite important, and the paper clearly shows how much graph side information helps under the assumed setting. It is known that side information helps but a clear quantification of its benefits is not there in existing literature.  On the negative side, it seems that the model and the method are too restrictive for the general rating estimation problem. Compared to general methods like Rao et al [42], this work seems too fine-tuned for the theoretical problem (with clustered users) and has limited direct impact on practice.  Originality: The theory and algorithm relies heavily on the recent literature on block models. The use of this theory in rating estimation is somewhat novel, although some recent works have made similar connections in related problems (for example, [R1]).  Significance and quality: The main positive for the paper is its significance. Quantifying the usefulness of side information in any learning problem is a difficult task. The paper achieves this goal by presenting and analysing a formal problem, which shows that the optimal sample complexity is indeed lower if side information is used.  The theory looks sound, but I have not read the proofs in the supplementary. Apart from the correctness, I have a fundamental question: - it seems that graph side information only aids when the quantity I_s is of the order log(n)/n. Doesn’t this imply that side information does not provide significant advantages? That is, we cannot drastically reduce the sample complexity by using side graphs (unless parameters are fine-tuned as in Example 2).  The practicality of the work is rather limited. The model assumes that users are clustered into two groups, the graph shows exactly these two groups, and the spectral algorithm mainly tries to find the groups. Hence, in comparison to general methods like Rao et al [42], this work seems too fine-tuned to the theoretical problem (with clustered users).  - can methods like [42] that simply use the graph as a regulariser achieve near optimality in this setting?  I like the fact that the proposed spectral method adaptively estimates some parameters, though I am not sure how c1, c2 can be estimated. However, as noted above, the methods loses generality as it mainly uses the graph for its cluster structure and not as some sort of regulariser (side graphs without community structure don't contribute).  The numerical section is bit disappointing, which is fine for a theory paper. The networks are real, but the ratings are synthetically generated based on the ground truth communities. - can we use the proposed method for any real binary estimation problem? - the numerical comparisons are perhaps bit unfair since the study is on a setup where finding the cluster structure from graph is supposed to reveal correct ratings, and hence, proposed method outperforms    Figure 2 is supposed to show that there is a clear phase transition (white to black) at the derived theoretical sample complexity (orange line). Given how convincing empirical phase transition plots can be for some problems, I am not sure if Figure 2 validates the sudden transition proved in Theorem 1. Please clarify.  Clarity: The paper is generally well-written. Since this is a theory paper in a general ML conference, it would be great if the authors can add some discussions on the practical consequences of the results. For example: - I_s of the order log(n)/n is only regime when side information helps. Why doesn’t it help to have more dense disjoint graphs? This is practically counter-intuitive. - when does it not matter to have side information? Why? - what is the implication of exact recovery in practice?  Minor comments: - line 67: “out-of-reach” is too strong, perhaps “not studied” - line 205: “cluster size” should be “number of clusters" - line 220: should w be \omega? - line 223: footnote mark 3 looks confusing after the (log n).  [R1] Lee and Shah. Reducing crowdsourcing to graphon estimation, statistically. AISTATS 2018  UPDATE AFTER AUTHOR REBUTTAL: I thank the authors for their response. It will be great if they can add the mentioned discussions to the paper. On a general note, I would also request the authors to include some concluding remarks about the limitation of the study to clustered user scenario, and if (or how) one can extend these results to more general graph side information.