Covariance shrinkage for autocorrelated data
The accurate estimation of covariance matrices is essential for many signal processing and machine learning algorithms. In high dimensional settings the sample covariance is known to perform poorly, hence regularization strategies such as analytic shrinkage of Ledoit/Wolf are applied. In the standard setting, i.i.d. data is assumed, however, in practice, time series typically exhibit strong autocorrelation structure, which introduces a pronounced estimation bias. Recent work by Sancetta has extended the shrinkage framework beyond i.i.d. data. We contribute in this work by showing that the Sancetta estimator, while being consistent in the high-dimensional limit, suffers from a high bias in finite sample sizes. We propose an alternative estimator, which is (1) unbiased, (2) less sensitive to hyperparameter choice and (3) yields superior performance in simulations on toy data and on a real world data set from an EEG-based Brain-Computer-Interfacing experiment.
840:this is a very nice paper, with compelling theoretical, simulated, and real data results.  i have a few majorish issues, and some minor ones.major-- one can choose lambda via CV or some theoretical tool.  if the theoretical tool has no parameters, it is a clear win.  however, there is a truncation parameter here. this manuscript did not convey to me *how* to choose b, and importantly, the extent to which the results are robust to this choice of b.  if this method is to be adopted as the de facto standard, some discussion about how to choose b and robustness to that choice is necessary.-- given that the main justification of using this method over CV is computational time, one might also acknowledge that practitioners always weigh a trade-off between accuracy and time.  clearly, this method is faster than CV, assuming we have a good way of choosing b.  but, how accurate is it? if it is much less accurate, than the improvement in time might not be so useful.  for example, in the real data example, we could simply use the average class covariances for the other subjects.  this would be fast, parameter free, and maybe just as accurate?minor-- in eq 5, b is some constant that satisfies some properties as a function of n? please clarify more formally the assumptions on b. also, please explain b.  please define the truncation kernel here.-- "we will provide a complementary analysis on the behaviour of the estimator for finite n."perhaps state a 'complementary theoretical analysis', i was led to believe you possibly meant only numerical, which of course, is much weaker.-- line 206, space missing-- remarks on thm 1: i would like more explanation of the relative size of the 3 biases.  the biases are a function of b, n, s and covariances.  some plotting showing the relative magnitude, say, of bias(San) vs bias(BC) would be very helpful.  for example, a heatmap showing bias(San)-bias(BC) for fixed n when varying b and s, or fixed function b_n and varying n & s.-- i don't understand the simulation setting.  please explain it more clearly, with equations, the notation for the 'parameter matrix' is unclear to me, what are '/' meant to denote? also, i don't know the abbreviation 'cmp'. if you are just trying to save space, i recommend removing some paragraph breaks, and keep content as clear as possible.-- a supplementary figure justifying footnote 4 is requested.-- "We average over R multivariate "  ok, what do you set R to be for these simulations?-- i think a better justification for *why* one would want to estimate a covariance matrix from an AR process, rather than the dynamics matrix, is in order.  in the end of the manuscript, you demonstrate an important application that totally justifies, but leading up to that, i was wondering.