Optimal Linear Estimation under Unknown Nonlinear Transform
Linear regression studies the problem of estimating a model parameter β
We consider a significant generalization in which the relationship between x i , β * and y i is noisy, quantized to a single bit, potentially nonlinear, noninvertible, as well as unknown. This model is known as the single-index model in statistics, and, among other things, it represents a significant generalization of one-bit compressed sensing. We propose a novel spectral-based estimation procedure and show that we can recover β * in settings (i.e., classes of link function f ) where previous algorithms fail. In general, our algorithm requires only very mild restrictions on the (unknown) functional relationship between y i and x i , β * . We also consider the high dimensional setting where β * is sparse, and introduce a two-stage nonconvex framework that addresses estimation challenges in high dimensional regimes where p n. For a broad class of link functions between x i , β * and y i , we establish minimax lower bounds that demonstrate the optimality of our estimators in both the classical and high dimensional regimes.
This paper concerns the problem of estimating a vector \beta, from +/-1 measurements y_i which depend statistically on linear functions < x_i, \beta >, where the x_i are Gaussian random vectors. The statistical relationship between y_i and < x_i, \beta > is captured through a link function P[ y_i = 1 | X_i = x_i ] = 1/2 f( < x_i, \beta > ) + 1/2. This model is general enough to capture compressed sensing and phase retrieval problems with binary measurements. The paper assumes that f is completely unknown ahead of time, but that it satisfies certain moment conditions. The paper shows how, under these moment conditions, to reduce the problem of estimating \beta to a sparse PCA problem, with covariance matrix generated by pairs of observations. The idea is to look at differences of x_i - x_{i'} and y_i - y_{i'}; the paper proves that the population covariance matrix of \delta_y \delta_x is a spiked identity matrix, where the spike is of the form \beta* \beta*^T. This clever reduction appears to be the main contribution of the work. The paper then proposes and analyzes algorithms for solving the resulting sparse PCA problem. In the ``low-dimensional'' setting in which n > p, it analyzes the power method, while in the ``high-dimensional'' setting with \beta sparse and n < < p, it analyzes a combination of SDP relaxation and truncated power method. The paper presents bounds on estimation error and a minimax lower bounds. The rates of convergence are optimal, once the number of samples is large (n > s^2 log p).  In the high-dimensional regime, the proposed approach requires s^2 log p samples to recover an s-sparse vector. This matches the best known sample complexity for sparse phase retrieval (which may be optimal for efficient algorithms - nobody currently knows). However, it is suboptimal for one-bit compressed sensing. This seems to be the main disadvantage of the proposed approach: for the sparse setting, there seems to be no way to use this reduction with less than s^2 samples; even though the problem might be solvable with about s samples by some other efficient algorithm. Because an improvement to the second stage of the algorithm would consist of a better algorithm for sparse PCA, this bottleneck seems insurmountable. Because of the requirement that n > s^2 log p, I feel the title overclaims a bit. Of course, if there was some result indicating that about s^2 samples are necessary to solve problems in this class (maybe by reduction *from* sparse PCA?), this would go a long way to addressing this issue.   The paper obtains a minimax lower bound of the correct order \sqrt{s log p / n}. This lower bound requires a gap condition on the link function f, such that f is bounded away from 1. This restriction may only be technical in nature, but it seems to rule out most of the interesting examples. For example, in noisy one-bit compressed sensing, where y_i = sign( < x_i, w > + z_i ), with z_i independent gaussian noise, the link function f(t) approaches one as t -> \infty. [As the paper comments, the minimax lower bound also does not apply to the noiseless version of one bit compressed sensing]. It would be helpful if the authors' response can specify some interesting settings which do satisfy the gap condition. There seems to be a minor notational issue in the theorems, as the algorithms can produce either an accurate estimate of \beta* or an accurate estimate of - \beta*. This ambiguity seems inevitable in this setting; please comment. The two stage algorithm for the sparse regime involves SDP and then a nonconvex refinement. The SDP initialization limits its applicability to problems in several thousands of dimensions. This may be find for machine learning problems, but for phase retrieval (where the dimensionality is the number of pixels in the image to be recovered), it is limiting. Is it possible to avoid the SDP initialization, e.g., by simply solving one nonconvex problem?