Low-rank Interaction with Sparse Additive Effects Model for Large Data Frames
Many applications of machine learning involve the analysis of large data framesmatrices collecting heterogeneous measurements (binary, numerical, counts, etc.) across samples -with missing values. Low-rank models, as studied by Udell  et al. [27], are popular in this framework for tasks such as visualization, clustering and missing value imputation. Yet, available methods with statistical guarantees and efficient optimization do not allow explicit modeling of main additive effects such as row and column, or covariate effects. In this paper, we introduce a lowrank interaction and sparse additive effects (LORIS) model which combines matrix regression on a dictionary and low-rank design, to estimate main effects and interactions simultaneously. We provide statistical guarantees in the form of upper bounds on the estimation error of both components. Then, we introduce a mixed coordinate gradient descent (MCGD) method which provably converges sub-linearly to an optimal solution and is computationally efficient for large scale data sets. We show on simulated and survey data that the method has a clear advantage over current practices.
Summary ------- This paper introduces a new statistical model for matrices of heterogeneous data (data frames) based on the exponential family. The features of this model are: i) modeling additive effects in a sparse way, ii) modeling low-rank interactions. The parameters of this model are then estimated by maximizing the likelihood with sparse and low-rank regularizations. In addition, this work comes with statistical guarantees and optimization convergence guarantees of the proposed algorithm. Numerical experiments concludes the manuscript.  Quality ------- This paper is mathematically rigorous and technically sound. Overall, this is a complete piece of work with theoretical guarantees (concerning the statistical aspect and the convergence of the proposed algorithm) and practical concerns (the estimation procedure is designed for large datasets). These two parts involve complex tools from statistics and optimization that seem completely understood by the authors.  In the problem formulation (Section 2), the Gaussian case assumes known (observed) variance. Isn't that limited? Moreover, there are many hypotheses needed for the statistical guarantees. Is it common? Is it really necessary?  The last section of the paper addresses two numerical experiments. The first one, on a synthetic dataset, honestly shows that the proposed method performs equally to a basic two-step procedure concerning estimating the low-rank parameter, but is beyond concerning the sparse parameter and the computational time (for large datasets). The second numerical experiment compares two versions of the proposed method (mixed effect versus Gaussian) about imputation error on a survey dataset.  In both cases, I find the numerical evaluation quite basic and limited. What about more advanced methods? Previous works? As for me, this is a flaw of this paper (detailed below): the link with the related work is unclear.  Clarity ------- This submission is a readable paper about a subject I am not familiar with. I find it very clear and polished (the only typos I saw is "match" L. 230 and \lambda_1 L. 157).  The authors made a noticeable effort to explain the implications of hypotheses (Section 2). But what about their concrete meaning? For instance, how restrictive is the last part of H1? What about positive definiteness of G in H3? Which distributions satisfy H4?  More importantly, I do not understand the aim of H5 since, by the model, Y_ij are assumed distributed according to an exponential family.  Moreover, Theorem 1 is not very clear to me: does it prove consistency? When \pi goes to 1, one expects the estimators to be consistent, but it is not clear with the proposed bounds.  Some references are missing L. 162 and 213.  Originality ----------- This work is a novel and smart combination of well-known techniques, concerning both statistics and optimization. However, it is not clear how it differs from previous studies. In particular, a section concerning related work is missing (the optimization part is briefly linked to prior work P. 6 but this effort is missing for the statistical part). I agree that the page limitation makes the thing difficult for the authors, yet the link to prior work is needed to assess the paper.  For instance, this submission seems quite supported by [23], but how it is related? Concretely, what is new in the statistical model (3)? This remark also holds for the (statistics and optimization) proofs: are they minor modifications of existing content or original work?  In addition, this lack also holds for the numerical section, where the proposed approach is only compared to either a basic procedure or itself.  Significance ------------ As for me, the work described in this submission is a significant contribution to the field of data frame analysis, both by its statistical and optimization aspects. It deserves publication as soon as the link to the prior work is made clearer and the numerical evaluation is strengthened by competitive advanced methods.  Rebuttal ----------- I have read the rebuttal and I thank the authors for their clear response. The authors agreed to add a "Related work" section, clarifying the link to prior work and originality of the proposed study. They also answered convincingly the reviewers' questions. Consequently, I have upgraded the overall score.