In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models. Yet, such approaches typically rely on approximate inference techniques such as variational inference which makes learning more complex and often less scalable due to approximation errors. We propose a new deep approach to Kalman filtering which can be learned directly in an end-to-end manner using backpropagation without additional approximations. Our approach uses a highdimensional factorized latent state representation for which the Kalman updates simplify to scalar operations and thus avoids hard to backpropagate, computationally heavy and potentially unstable matrix inversions. Moreover, we use locally linear dynamic models to efficiently propagate the latent state to the next time step. While our locally linear modelling and factorization assumptions are in general not true for the original low-dimensional state space of the system, the network finds a high-dimensional latent space where these assumptions hold to perform efficient inference. This state representation is learned jointly with the transition and noise models. The resulting network architecture, which we call Recurrent Kalman Network (RKN), can be used for any time-series data, similar to a LSTM (Hochreiter and Schmidhuber, 1997) but uses an explicit representation of uncertainty. As shown by our experiments, the RKN obtains much more accurate uncertainty estimates than an LSTM or Gated Recurrent Units (GRUs) (Cho et al., 2014) while also showing a slightly improved prediction performance and outperforms various recent generative models on an image imputation task.
This paper proposes, Recurrent Kalman Network, a modified Kaman filter in which the latent dynamics is projected into a higher dimensional space; efficient inference in this high-dimensional latent space is possible due to the space being locally linear. The state representation, transition, and observation models are learned jointly by backpropagation.  
The paper is well written and the model is clearly explained; I also like the simplicity of the idea that uses the same machinery of Kalman filter.  However, I believe the authors can improve the presentation of the model and empirical evaluation. 
In terms of model presentation, the authors can compare the model with a large set of deep recurrent models that have recently been proposed for modeling time series with nonlinear latent dynamics (e.g. Variational Sequential Monte Carlo, Structured inference networks for nonlinear state space models, Black box variational inference for state space models, Composing graphical models with neural networks for structured representations and fast inference, etc.). For instance, a table of some of these models with their pros and cons can be helpful for guiding the reader.
In terms of model evaluation, the paper needs a better evaluation section specifically on the generative models (see examples above) that are much  more suitable for modeling uncertainty compared to LSTM/GRU. More specifically, another approach for alleviating the limitations of Kalman filter would be to use non-linear transitions based on some non-linear functions approximation. This approach has been proposed in deep Kaman filter (Krishnan 2015) and it would be interesting to see how well your model performs compared to that for modeling uncertainty and computing predictive log-likelihood. 
In conclusion, I think the paper presents a nice idea but it requires more work in order to pass the ICLR acceptance threshold.
----------------------------------------------------------
The authors have addressed my comments and as a result I changed my rating to 6.