Evaluating the statistical significance of biclusters
Biclustering (also known as submatrix localization) is a problem of high practical relevance in exploratory analysis of high-dimensional data. We develop a framework for performing statistical inference on biclusters found by score-based algorithms. Since the bicluster was selected in a data dependent manner by a biclustering or localization algorithm, this is a form of selective inference. Our framework gives exact (non-asymptotic) confidence intervals and p-values for the significance of the selected biclusters.
The authors provide a framework for constructing p-values for the output of a variety of "selection" tasks, focusing on biclusters in particular. Their approach is to characterize the distribution of test statistics as multivariate normal random variables constrained to linear polytopes. This framework allows them to construct exact tests (assuming normality) for subsets of variables that have been selected in different ways.Quality: This is a quality paper. The contribution is non-trivial, and the material is presented in a way that goes beyond solving "just the bicluster significance problem." (Which is important in and of itself.) Clarity: Good but I have suggestions for improvement; see below. Originality: I believe that the specialization of theorem from reference [10] to biclustering and scan statistics and the rates that go with are novel. Significance: Addressing statistical significance issues in this domain is very important and widely applicable.-Specific comments:028: '"significant" compared to the rest of X. 063: "...statistically significant bicluster." 088: "We ask whether a submatrix selected by a biclustering algorithm captures the hidden association(s)." I think captures "some" of the hidden association(s) would be more accurate.The inference is that there is *something* in the computed bicluster that has some signal. In particular, supposing that rows correspond to genes, the specified alpha level does not control false discovery of genes within the computed bicluster; it controls false discovery of an "empty" bicluster. (Similarly, "power" does not indicate the probability of having found the entire bicluster, but only a part of it.) I think it is extremely important to make this clear. If on the other hand I am mistaken about this, I think other readers would be as well.039: Z ~ N(...) I have to ask the cliched question, what about non-normal noise models? How sensitive is the procedure to different kinds of non-normality? Depends on k maybe?071: "...for the amount of the signal in..." I suggest explicitly defining the selected parameter, maybe before (1.2), so that you can state this formally. I also think it would be worth re-iterating that the procedure will trap the *selected* parameter with probability (1-\alpha), where the procedure is to run the bi-clustering algorithm, select the parameter of interest, and then compute/invert the p-value.077: "In the supplementary materials, we discuss the problem in the more general setting where there are multiple emnbedded [SIC -- fix this] submatrices..." I recommend finding a way to lift this discussion into the main paper, even if you can't find room to give the full details. Not saying that needs to go here exactly, but somewhere in the main paper.129: Use either the e^TXe form or the trace form, but choose one and use it consistently.150: "The amount of signal in..." See comment above.162: "Let eta ... a linear function of y" The "linear function" comment confused me; just let it be a vector.168: I assume V0 is relevant in other contexts; if it's not relevant for this work perhaps omit?162--174: There is a lot of notation overloading here that is made worse by the fact that vectors are typeset indistinguishably from scalars. In particular, see if there is a way to remove overloading in arguments of F (2.7).378--397: These plots are difficult to distinguish, and I'm not sure what the intended message is. If it is to compare power of random initialization against intelligent initialization, then they should appear on the same plot. Then have different plots for (e.g.) 3 different n.