Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models. Yet, such approaches typically rely on approximate inference techniques such as variational inference which makes learning more complex and often less scalable due to approximation errors. We propose a new deep approach to Kalman filtering which can be learned directly in an end-to-end manner using backpropagation without additional approximations. Our approach uses a high-dimensional factorized latent state representation for which the Kalman updates simplify to scalar operations and thus avoids hard to backpropagate, computationally heavy and potentially unstable matrix inversions. Moreover, we use locally linear dynamic models to efficiently propagate the latent state to the next time step. While our locally linear modelling and factorization assumptions are in general not true for the original low-dimensional state space of the system, the network finds a high-dimensional latent space where these assumptions hold to perform efficient inference. This state representation is learned jointly with the transition and noise models. The resulting network architecture, which we call Recurrent Kalman Network (RKN), can be used for any time-series data, similar to a LSTM (Hochreiter and Schmidhuber, 1997) but uses an explicit representation of uncertainty. As shown by our experiments, the RKN obtains much more accurate uncertainty estimates than an LSTM or Gated Recurrent Units (GRUs) (Cho et al., 2014) while also showing a slightly improved prediction performance and outperforms various recent generative models on an image imputation task.
 This paper presents a particular architecture for a probabilistic recurrent neural network that is based on ideas from Kalman filtering. Whereas Kalman filters are used to infer the state of a known generative model (a linear-Gaussian dynamical system), here, the authors jointly learn a recursive filter without explicitly formulating a generative model of the data. The paper deals with an important problem and the approach has many appealing characteristics: it learns a state representation and its associated transition dynamics, it learns nonlinear filter that can be used online and it learns encoders/decoders from high-dimensional observations to the state. The article does not provide any probability density (even though learning happens by maximizing a likelihood) and there are no connections to probabilistic generative models. In my opinion this is a pity since this would shed more light into the characteristics of the proposed approach. I believe that the model could be presented more clearly. For example, the Preliminaries section uses formulas before defining them. Also, explicitly writing the high-level chain of computations from o_t and z_{t-1}^+ to o_t^+ and s_t^+ would be extremely useful. Even more than Fig. 1, in my opinion. All in all, I have found this an interesting architecture for a RNN but would have appreciated more insight into its relationships with the large body of generative probabilistic state-space models and the methods to perform inference on them.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models. Yet, such approaches typically rely on approximate inference techniques such as variational inference which makes learning more complex and often less scalable due to approximation errors. We propose a new deep approach to Kalman filtering which can be learned directly in an end-to-end manner using backpropagation without additional approximations. Our approach uses a high-dimensional factorized latent state representation for which the Kalman updates simplify to scalar operations and thus avoids hard to backpropagate, computationally heavy and potentially unstable matrix inversions. Moreover, we use locally linear dynamic models to efficiently propagate the latent state to the next time step. While our locally linear modelling and factorization assumptions are in general not true for the original low-dimensional state space of the system, the network finds a high-dimensional latent space where these assumptions hold to perform efficient inference. This state representation is learned jointly with the transition and noise models. The resulting network architecture, which we call Recurrent Kalman Network (RKN), can be used for any time-series data, similar to a LSTM (Hochreiter and Schmidhuber, 1997) but uses an explicit representation of uncertainty. As shown by our experiments, the RKN obtains much more accurate uncertainty estimates than an LSTM or Gated Recurrent Units (GRUs) (Cho et al., 2014) while also showing a slightly improved prediction performance and outperforms various recent generative models on an image imputation task.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents a particular architecture for a probabilistic recurrent neural network that is based on ideas from Kalman filtering. Whereas Kalman filters are used to infer the state of a known generative model (a linear-Gaussian dynamical system), here, the authors jointly learn a recursive filter without explicitly formulating a generative model of the data. The paper deals with an important problem and the approach has many appealing characteristics: it learns a state representation and its associated transition dynamics, it learns nonlinear filter that can be used online and it learns encoders/decoders from high-dimensional observations to the state. The article does not provide any probability density (even though learning happens by maximizing a likelihood) and there are no connections to probabilistic generative models. In my opinion this is a pity since this would shed more light into the characteristics of the proposed approach. I believe that the model could be presented more clearly. For example, the Preliminaries section uses formulas before defining them. Also, explicitly writing the high-level chain of computations from o_t and z_{t-1}^+ to o_t^+ and s_t^+ would be extremely useful. Even more than Fig. 1, in my opinion. All in all, I have found this an interesting architecture for a RNN but would have appreciated more insight into its relationships with the large body of generative probabilistic state-space models and the methods to perform inference on them.
assistant
 This paper presents a particular architecture for a probabilistic recurrent neural network that is based on ideas from Kalman filtering. Whereas Kalman filters are used to infer the state of a known generative model (a linear-Gaussian dynamical system), here, the authors jointly learn a recursive filter without explicitly formulating a generative model of the data. The paper deals with an important problem and the approach has many appealing characteristics: it learns a state representation and its associated transition dynamics, it learns nonlinear filter that can be used online and it learns encoders/decoders from high-dimensional observations to the state. The article does not provide any probability density (even though learning happens by maximizing a likelihood) and there are no connections to probabilistic generative models. In my opinion this is a pity since this would shed more light into the characteristics of the proposed approach. I believe that the model could be presented more clearly. For example, the Preliminaries section uses formulas before defining them. Also, explicitly writing the high-level chain of computations from o_t and z_{t-1}^+ to o_t^+ and s_t^+ would be extremely useful. Even more than Fig. 1, in my opinion. All in all, I have found this an interesting architecture for a RNN but would have appreciated more insight into its relationships with the large body of generative probabilistic state-space models and the methods to perform inference on them.