Estimating Jaccard Index with Missing Observations: A Matrix Calibration Approach
The Jaccard index is a standard statistics for comparing the pairwise similarity between data samples. This paper investigates the problem of estimating a Jaccard index matrix when there are missing observations in data samples. Starting from a Jaccard index matrix approximated from the incomplete data, our method calibrates the matrix to meet the requirement of positive semi-definiteness and other constraints, through a simple alternating projection algorithm. Compared with conventional approaches that estimate the similarity matrix based on the imputed data, our method has a strong advantage in that the calibrated matrix is guaranteed to be closer to the unknown ground truth in the Frobenius norm than the un-calibrated matrix (except in special cases they are identical). We carried out a series of empirical experiments and the results confirmed our theoretical justification. The evaluation also reported significantly improved results in real learning tasks on benchmark datasets.
The paper was clear and concise. The idea is original as far as the reviewer is aware, and after a modest amount of literature search. The experimental results are convincing enough to warrant considering the approach for small to medium sized problems, with sample sizes in the order of 1K-10K, and the ratio of observed features is in the .2-.5 range. It's not clear from the experimental evaluation, however, whether the approach is as effective, compared to the other imputation approaches, on larger problems (larger similarity matrices) when projecting onto the set of positive semi-definite matrices - which relies on computation of SVD - becomes intractable or requires approximate methods (the authors do not clarify whether approximate SVD was used in the experiments). As the authors point out, the matrix decomposition is a major computational bottleneck and limitation of this method, and as such the reviewer would argue that this would make it an unappealing method to apply to many real world problems where sample sizes are much larger that 10K. This (what happens with larger data sets) is a major concern. Another concern is the relevance of this whole apparatus. I am not aware, off the cuff, of a practical situation in which one would want to compute Jaccard similarity matrix while facing missing values in the samples. It is quite possible that such a situation comes up, or even is very important, but the authors fail to provide any practical examples. All of the experiments are on data sets (like MNIST or USPS) in which the values are dropped artificially. Obviously, such experiments are crucial to evaluate the accuracy of the proposed method under controlled circumstances, and to measure its sensitivity to various parameters. But equally crucial, I think, is to provide at least one example in which real data, with real underlying phenomenon of interest (here the missing values) is examined. Since in real data missing values are most likely not distributed uniformly (see below more on this) it would be potentially a different "game" and the outcome, in particular the relative standing of the proposed method vs. imputation, is uncertain. I'd welcome report on any additional experiment along these lines that the authors can include in the rebuttal.Another suggestion for improvement, related to the comment above: consider cases other than uniformly random missing features. At some point in continuum from high to low entropy distributions one could expect imputation via kNN to outperform.