Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization
Stochastic optimization naturally arises in machine learning. Efficient algorithms with provable guarantees, however, are still largely missing, when the objective function is nonconvex and the data points are dependent. This paper studies this fundamental challenge through a streaming PCA problem for stationary time series data. Specifically, our goal is to estimate the principle component of time series data with respect to the covariance matrix of the stationary distribution. Computationally, we propose a variant of Oja's algorithm combined with downsampling to control the bias of the stochastic gradient caused by the data dependency. Theoretically, we quantify the uncertainty of our proposed stochastic algorithm based on diffusion approximations. This allows us to prove the asymptotic rate of convergence and further implies near optimal asymptotic sample complexity. Numerical experiments are provided to support our analysis.
Summary: The paper consider the setting of streaming PCA for time series data which contains two challenging ingredients: data stream dependence and a non-convex optimization manifold. The authors address this setting via downsampled version of Oja's algorithm. By closely inspecting the optimization manifold and using tools from the theory of stochastic differential equations, the authors provide a rather detailed analysis of the convergence behavior, along with confirming experiments on synthetic and real data.  Evaluation: Streaming PCA is a fundamental setting in a topic which becomes increasingly important for the ML community, namely, time series analysis. Both data dependence and non-convex optimization are still at their anecdotal preliminary stage, and the algorithm and the analysis provided in the paper form an  interesting contribution in this respect. The paper is well-structured, although I find some key definitions missing, and some technical parts somewhat hard-to-follow.   Comments: L47 - I find 'Moreover, theses results..' somewhat vague. Can you further elaborate on the nature of the uncertainty not captured by current convergence analyses from your point of view? L115 - Many terms here are used without providing a definition first. E.g., $\mathcal{F}$ in $(\mathcal{S},\mathcal{f})$, positive recurrent, transition kernel,.. L162 - 'The eigengaps in ... is identfible' seems to have a grammatical mistake. L169 - Can you put your statements regarding the taxonomy of stationary points on a more rigorous ground? L274 - Most of the proof sketch addresses the continuous analog of the discrete dynamic, i.e., (stochastic) gradient flow. The derivation of the convergence rate for the original discrete setting is less clear to me. Also, I would suggest adding an explicit statement covering the conditions and convergence rates implied by the analysis, at earlier stages of the paper before the proof sketch (as well as other contributions made by the paper - such as the intimate analysis of the algorithm at different stage of the optimization process using SDE).