Learning Theory and Algorithms for Forecasting Non-Stationary Time Series
We present data-dependent learning bounds for the general scenario of nonstationary non-mixing stochastic processes. Our learning guarantees are expressed in terms of a data-dependent measure of sequential complexity and a discrepancy measure that can be estimated from data under some mild assumptions. We use our learning bounds to devise new algorithms for non-stationary time series forecasting for which we report some preliminary experimental results.
Summary:This paper studies learning with time series with quite relaxed assumptions on the stochastic process defining the time series: No mixing or stationarity assumptions is required.This is achieved by the use of two concepts:1) Sequential covering number, which roughly speaking is an extension of the standard covering number to a norm defined over binary tree structures, and 2) Discrepancy, which is a measure of the non-stationarity of the process.The paper provides a high probability deviation inequality that is a function of the covering number and discrepancy. It suggests a method for estimating the discrepancy, under some extra assumptions. It also introduces some algorithms for forecasting using the obtained upper bound.Even though having an upper bound that is valid for non-stationary and non-mixing processes might seem almost magical, one might say that the trick is that one can move all the apparent complexity of the process into the discrepancy and the sequential covering number. If the process is too difficult to learn, these quantities would be too large.The interesting point here, however, is that the quantities that measure the complexity of learning are defined at the right level. To understand my comment, notice that the usual approach to study learning for time series is that we start by analyzing the difficulty of stochastic process (through concepts such as mixing) and then under certain assumptions on the stochastic process, we convert it to the usual i.i.d. setting using techniques such as independent blocks. One can then define covering number/Rademacher complexity/etc. over these independent blocks, and use the standard tools that work for independent samples. This approach separates the study of complexity of the stochastic process and the complexity of function/hypothesis space, and may lead to pessimistic results.What this paper does is that it defines the complexity at the joint stochastic process-function space level. This can potentially lead to much tighter results.The key elements of this approach are not necessarily new.Sequential complexities have been defined by Rakhlin, Sridharan, Tewari (NIPS 2010, NIPS 2011, JMLR 2015). A similar discrepancy measure has also been defined and used by Kuznetsov and Mohri (ALT 2014), Mohri and Munoz Medina (ALT 2012), and even before that in the domain adaptation literature.The use of these together, however, is novel as far as I know.  Evaluation:The topic of the paper is interesting and important, the approach is elegant and as far I can tell novel, and the paper itself is more or less well-written. I have some comments below and I encourage the authors to answer and/or consider them. * L142: Is q_t missing in the LHS?* Is optimizing only the discrepancy term of the upper bound of Corollary 3 the right thing to do?Corollary 2 shows that the upper bound depends on the choice of q. In the beginning of Section 4, it is suggested that one should minimize the discrepancy as a function of q and using it to weigh the training samples.If I understand correctly, the covering number also depends on q (as the cover is defined w.r.t. q-weighted l_1 norm). If that is the case, shouldn't it be considered in the optimization too? (I understand that this might be computationally very difficult or impossible.)* Please compare your algorithms with those in Kuznetsov and Mohri, "Forecasting Non-Stationary Time Series: From Theory to Algorithms," NIPS 2014 Workshop on Transfer and Multi-task Learning. Montreal, Canada, December 2014.* Is it possible to have Bernstein-like bound in Theorem 1?* L222: It seems that the first inequality can actually be an equality.* L246: It might be more clear to use c_t(z_t(\sigma)) instead of c_t(\sigma).* L251: exp(lambda alpha) -> exp( 2 lambda alpha)* L256: c_T(c) -> c_t(sigma)* Some examples of the behaviour of E[N_1(alpha,G,v)] would be helpful. For example, if the process has a certain mixing property, is it different from when it is i.i.d.?* L137: "definition definition" -> "definition"* L177: "leg" -> "lag"* L339-342: The discussion is a bit vague here.* L350: delta' -> delta* L492 (Appendix A): Maybe it is more clear to say that (z,z') is from T(p).* L598: Isn't the change from q to the uniform u too loose? This leads to M || q - u ||_1, which kind of ignores all the structure in the function space except its boundedness.