Closed-form Estimators for High-dimensional Generalized Linear Models
We propose a class of closed-form estimators for GLMs under high-dimensional sampling regimes. Our class of estimators is based on deriving closed-form variants of the vanilla unregularized MLE but which are (a) well-defined even under high-dimensional settings, and (b) available in closed-form. We then perform thresholding operations on this MLE variant to obtain our class of estimators. We derive a unified statistical analysis of our class of estimators, and show that it enjoys strong statistical guarantees in both parameter error as well as variable selection, that surprisingly match those of the more complex regularized GLM MLEs, even while our closed-form estimators are computationally much simpler. We derive instantiations of our class of closed-form estimators, as well as corollaries of our general theorem, for the special cases of logistic, exponential and Poisson regression models. We corroborate the surprising statistical and computational performance of our class of estimators via extensive simulations.
- Summary of Paper   - The paper approaches the problem of estimating parameters in GLMs     when the dimension of the data is larger than the number of     samples. The authors claim to have a closed-form solution for     such a setting based on thresholding and projection operations.     The closed-form estimators enjoy some favorable frequentist     statistical properties.Finally, the authors compare their     estimates to l1 regularized MLE estimates for two simulated data     sets. - Quality   - I have questions (L120 comments below) about the work that trace     back to the way the authors formulated the exponential family     form of logistic regression and poisson regression. This issue     impacts how the authors define the subset of the interior of the     mean-response polytope and therefore the evidence provided in the     results section.   - L120: The authors' definition of the logistic regression is     confusing and I'm not sure if it is correct. I might write the     exponential family as $p(y_i | theta, x) = exp( y__i * \eta -     A(\eta)$, where $\eta = log(\mu/(1-\mu))$ is the log-odds, the     log-partition function is $A(\eta) = log(1+\exp(\eta)$, and $\mu     = (1+\exp(\theta*x))^{-1}$ is the logistic function. This     formulation is very different than the one presented: $\exp{y*z -     \log(\exp{z}+\exp{-z}}$ where z = < \theta, x > is their natural     parameter. In particular the stated log partition function is     A(a) = log(exp(-a) + exp(a)). We could write that as A(a) =     log(exp(-a^2)). How did the authors arrive at the stated form for     the logistic regression exponential family density function?   - L120: I am similarly confused by how the authors arrived at their     exponential family formulation for the poisson regression     density. How did the authors arrive at this formulation and can     they show that it is certainly correct?   - L296: The theorem seems to provide for not missing an important     predictor with high probability, but we would also like to make     sure to set coefficient to zero that should be. Is this theorem     one-sided in that it only provides for not missing a non-zero     coefficient?   - L237: The article develops reasons for truncating (X^T * X) and     for the projection on the subset of the polytope in the     estimator. But the reason for the soft-thresholdS_{\lambda_n}     is not clear. This should be clarified.   - L296: The theorem shows that the estimator includes all non-zero     supports with a probability bound. Is this bound non-trivial in     any specific instances? - Clarity   - The paper is fairly clear, though dense. A few edits outlined     below could make it more clear.   - L041: The introductory paragraph starts with talking about     estimation and ends with talking about prediction. While these     concepts are related, it would help to be consisent so early in     the manuscript.   - L046: References would help after "and others"   - L106: The notation for the natural parameter in the exponential     family model < \theta, x > is non-standard and confusing. The     bracket notation is used by some to denote dot-product and other     to denote expected value. A more familar notation or a     justification for this particular notation would help the article     clarity.   - L118: The use of "A(a)" is confusing and "a" could be changed for     the authors' notation for the natural parameter < \theta, x >.   - L190: The article seems to rely heavily on \hat{M} the subset of     the interior of the response moment polytope. But we don't get to     see what that is until we get to specific models. It would help     to remind the reader that \hat{M} will be defined for specific     models more frequently.   - L194: The estimator is stated without development. Either put the     estimator after the development or state that development will     follow. The "machinery" is not described, it is the notation that     is described earlier. The "machinery" seems to be described     later. The reason for "Elem" subscript on \theta is not described.   - L322: Remark 3 refers to "k". I could not find what "k" is     earlier in the paper. - Originality   - The paper builds on some previous work on linear regression, but     the extension to the GLM setting is novel. - Significance   - As the authors state, even the MLE in a low-dimensional setting p     < n does not have a closed-form solution. So, the development of     a GLM estimator that is both closed form and valid in high     dimensional settings is highly significant.