Spectral Learning of Dynamic Systems from Nonequilibrium Data
Observable operator models (OOMs) and related models are one of the most important and powerful tools for modeling and analyzing stochastic systems. They exactly describe dynamics of finite-rank systems and can be efficiently and consistently estimated through spectral learning under the assumption of identically distributed data. In this paper, we investigate the properties of spectral learning without this assumption due to the requirements of analyzing large-time scale systems, and show that the equilibrium dynamics of a system can be extracted from nonequilibrium observation data by imposing an equilibrium constraint. In addition, we propose a binless extension of spectral learning for continuous data. In comparison with the other continuous-valued spectral algorithms, the binless algorithm can achieve consistent estimation of equilibrium dynamics with only linear complexity.
This paper concerns the modeling of discrete- and continuous-valued stochastic processes with observable operator models (OOMs), which (together with a number of closely related models like predictive state representations and multiplicity automata) offer fast constructive and asymptotically correct learning algorithms for a model class that properly include HMMs. Specifically, this article focusses on nonstationary processes. Technical contributions: 1. proofs that the standard model estimation techniques generalize immediately from stationary to nonstationary data (Section 3). 2. An algorithm to reconstruct (an estimate of) the asymptotic stationary process dynamics from initial nonstationary data sequences (Section 4). 3. A method, called "binless learning" or "binless OOMs" to estimate from continuous-valued initial nonstationary training sequences a novel kind of descriptive model of asymptotic characteristics of the process' distribution (Section 5). 4. Brief demos of the methods from Section 5 on synthetic data and data derived from simulations of an analytical model of a protein state switching dynamics. In these demos the objective is to estimate asymptotic process properties from training data that start from (very) out-of-equilibrium conditions. These demos reveal a much better compensation for the initial nonstationarity of training data than more traditional HMM models. 
Novelty and relevance  The analytical results presented in Section 3 are not new (see my more detailed comments below), though they probably have not yet been presented in exactly this form. I believe that the authors, who apparently come from a modeling community in the biosciences, simply are not aware that it is commonplace wisdom in the OOM field that OOM models of nonstationary processes can be estimated in the same way as stationary ones (as long as the nonstationarity can be described as the effect of a nonequilibrium starting state; stronger forms of nonstationarity where the very observable operators vary with time are not considered in this paper, and have only superficially been investigated by OOM researchers elsewhere). The core contribution of Section 4, an algorithm to recover the stationary state from an OOM estimated from nonstationary data, is novel (but I think it might be unnecessarily complicated, see my detailed comment below). The "binless learning" of "binless OOMs" in Section 5 is entirely novel and quite ingeneous in my view - for me the absolute bummer part in this paper. Judging from the first of the three demos in Section 6, truly amazing inferences can be drawn about asymptotic properties of a stochastic process from highly nonstationary initial-condition-plagued training data (while the remaining two demos I couldn't understand because they require previous knowledge about techniques and terminology from statistical chemistry (?) that is not explained and that I can't summon). Although I am not familiar with the kind of bio/chemistry microphysics applications that the authors use for display, it would think that these binless OOMs constitute an truly impressive and important step of progress.  Technical correctness  Apart from some minor lapses, mostly notational typos (pointed out below) - fine.   Presentation  Very good English, crisp command of mathematical formalism. I find that the Introduction over-emphasizes the challenges arising from nonequilibrium data -- these are really solved and easy issues as long as the nonstationarity can be captured by non-equilibrium starting state (the case considered in this article). In contrast, the Intro under-sells in my view the novel modeling opportunities afforded by binless OOM learning.   Summary assessment and recommendation  A very interesting and innovative contribution -- namely, in Section 5, concerning "binless OOMs". The novelty of Sections 3-4 is marginal, the authors are probably not aware of the existing common understanding of the matters discussed in these Sections (the novel algorithm in Appendix A.4 excepted). If this were a journal paper submission I would recommend a substantial revision with a much deflated treatment of the Section 3-4 material and a much more prominent role of the binless idea. The last two demos are useless because they are unintelligible due to compression - in a revision they would need to be expanded.      ********** more detailed comments **********   Lines 28 ff: the article presents the estimation of OOMs from nonstationary data as something that has been rarely done before ("a major challenge … In most literature the observation data are assumed to be equilibrium…"). That is not true. Even in the earliest OOM publications (H. Jaeger(1998): Observable operator models of stochastic processes: a tutorial. GMD Report 42, German National Research Center for Information Technology 1998) it was explicitly recognized that the standard OOM estimation procedure works for stationary and nonstationary data alike. I dare say that this is and always has been obvious to researchers active in this field. In the recent review paper of Thon (Thon, M., Jaeger, H. (2015): Links Between Multiplicity Automata, Observable Operator Models and Predictive State Representations -- a Unified Learning Framework. Journal of Machine Learning Research 16, 103-147) the assumption of stationarity is not even introduced. In other words, Theorem 1 cannot be considered new. Only the generalization 1 (line 129) adds a new aspect to the known picture.    Section 4: the initial text paragraph confuses me. It starts by saying that "the only remaining problem for learning OOMs from nonequilibrium data is how to estimate initial state vectors". But then the text proceeds to state that this problem is uninteresting and instead one should try to find the equilibrium state. I am further confused by the fact that in line 149 it is not explained what \hat{omega} is intended to be - is it intended to be the equilibrium state? then it should be written \hat{\bar{\omega}}. -- On a side note, it is confusing that the index _{eq} is used to denote an 'equivalent' OOM, and the \bar to denote an equilibrium. Both words start with 'eq' and in fact I lost one hour getting myself un-trapped from this duplicity. Please consider changing the notation.   Lines 148 ff: I don't understand why the equilibrium state \hat{\omega} (or maybe in better notation \hat{\bar{\omega}}) isn't simply calculated as an eigenvector of \hat{Xi}(\mathcal{O}) of largest absolute eigenvalue. Furthermore, some thought should be given to processes that have more than one ergodic component. Then there is no uniquely defined equilibrium state.   Section 5: the choice of terminology "binless learning" or "binless OOM" should be explained/motivated. I never heard of this concept before and it is fun to see what Google queries try to retrieve (zero hits on these key phrase, alternative suggestions: "boneless ham", "boneless BBQ"…).   The cover story for motivation the material from Section 5 should be phrased more carefully. Text in the Introduction suggests an alternative to parametric / Hilbert space embedding OOMs, but Section 5 only offers a model of asymptotic characteristics of a process, not a model of the process, so the juxtaposition to those continuous-valued OOM techniques is misleading. Introduction should be rephrased carefully.    It took me a little effort to understand the mathematical type of the object defined in Eqn (19). It would help the reader to explicitly state that this is a (point) probability measure on \mathcal{O} \times \mathbb{R}^D_1 \times \mathbb{R}^D_2.    Line 173f: "where feature functions can be selected as splines, radial basis functions or other …": this needs more explanation. How can e.g.  a spline function, applied to L points, give a real number as a feature value?  Simulation studies reported in Figs 3 and 4: impossible to understand in this super-condensed presentation. Readers are required to know what TIC analysis is and what w_{TICA} is. The NIPS readership will be lost. Please invest an explanatory paragraph.  ********* minor comments / edits *********  line 90, first formal expression, counter index should range over n not s  Figure 1: indices in circles should read t-L etc, not t-s  Assumption 2 page 4: please specify the range of the f_t functions. I assume it's \mathbb{R}? (or re-use notation \phi from section 3.1)  It would be helpful for the reader to state at some early point that nonstationarity of a rank-m process represented by an m-dimensional OOM is equivalent to \omega \neq  \omega_{eq} \Xi(\mathcal{O}).   line 149 typo "quadrantic"  line 192: principle -> principal  line 198: "for general g with" incomplete or "with" superfluous