One-shot learning and big data with n = 2
We model a "one-shot learning" situation, where very few observations y 1 , ..., y n ∈ R are available. Associated with each observation y i is a very highdimensional vector x i ∈ R d , which provides context for y i and enables us to predict subsequent observations, given their own context. One of the salient features of our analysis is that the problems studied here are easier when the dimension of x i is large; in other words, prediction becomes easier when more context is provided. The proposed methodology is a variant of principal component regression (PCR). Our rigorous analysis sheds new light on PCR. For instance, we show that classical PCR estimators may be inconsistent in the specified setting, unless they are multiplied by a scalar c > 1; that is, unless the classical estimator is expanded. This expansion phenomenon appears to be somewhat novel and contrasts with shrinkage methods (c < 1), which are far more common in big data analyses.
Update: The point of my simple method was to 
      obtain a better understanding of the theoretical analysis. I ran the same 
      experiments the authors ran from section 7, with d=500,5000,500000 and the 
      simple method began to converge to the oracle method and at d=500 it 
      certainly did not have an error greater than 500 (roughly 7). I ran all of 
      the experiments with n=2. **edit** The focus of these experiments were on 
      weak convergence as that was the theoretical analysis performed for n=2. 
      For n=2 the simple method does perform very poorly for squared loss, but 
      performs better for absolute loss. **end edit** I believe that 
      this one-shot idea is very interesting, but the bias correction has not 
      been thoroughly explored. The experiments are show a lot of promise, which 
      is why I remain borderline with this paper. ==== *** 
      This paper aims to provide an analysis for principle component 
      regression in the setting where the feature vectors $x$. The authors 
      let $x = v + e$ where $e$ is some corruption of the nominal feature 
      vector $v$; and $v = a u$ where $a \sim N(0,\eta^2 \gamma^2 d)$ while 
      the observations $y = \theta/(\gamma \sqrt{d}) \langle v,u \rangle + 
      \xi$. This formulation is slightly different than the standard one 
      because our design vectors are noisy, which can pose challenges in 
      identifying the linear relationship between $x$ and $y$. Thus, using 
      the top principle components of $x$ is a standard method used in order 
      to help regularize the estimation. The paper is relevant to the ML 
      community. The key message of using a bias-corrected estimate of $y$ 
      is interesting, but not necessarily new. Handling bias in regularized 
      methods is a common problem (cf. Regularization and variable selection 
      via the Elastic Net, Zou and Hastie, 2005). The authors present 
      theoretical analysis to justify their results. I find the paper 
      interesting; however I am not sure if the number of new results and 
      level of insights warrants acceptance. *** The result is 
      interesting, but lacks enough depth to provide a convincing argument 
      for its use. The authors present a very specific setting for the 
      design vectors: spiked covariance. The motivation of the bias 
      correction is interesting, but how do the authors expect it should 
      work under other models for $x$ and $y$? A simulation study of these 
      methods could prove useful in order to provide insights into the 
      behavior in the non-Gaussian setting. Furthermore, as the authors 
      admit, the signal of $x$ is very strong in the direction of the 
      regression vectors. It is conceivable that the signal $x$ is strong in 
      multiple directions (especially in the high-dimensional setting). How 
      would the bias-correction change when adapting to multiple PCA 
      factors? *** Overall, the model being used to perform the 
      analysis seems to simple to shed serious light on the problem. For 
      example, one can show risk consistency in the simple setting of 
      letting the largest principle component simply be equal $x_1$. That 
      is, only perform PCA one the first example and regression on the 
      second. The authors go through a lot of analysis from fundamentals 
      without providing intuition for why the shrinkage occurs. In 
      particular, the authors should explain why least squares (which 
      produces consistent estimators) must be corrected in this context. 
      Interestingly, the simple estimator of just taking $x_1$ to be the 
      principle component and performing regression on the remaining 
      examples performs better. Thus, rather than a shrinkage correction, a 
      more reasonable solution seems to be to learn the PC vectors on one 
      set of data and regress on the other. Can the authors provide more 
      intuition for why they expect their proposed method to perform better 
      in generic settings?