Temporal Regularization in Markov Decision Process
Several applications of Reinforcement Learning suffer from instability due to high variance. This is especially prevalent in high dimensional domains. Regularization is a commonly used technique in machine learning to reduce variance, at the cost of introducing some bias. Most existing regularization techniques focus on spatial (perceptual) regularization. Yet in reinforcement learning, due to the nature of the Bellman equation, there is an opportunity to also exploit temporal regularization based on smoothness in value estimates over trajectories. This paper explores a class of methods for temporal regularization. We formally characterize the bias induced by this technique using Markov chain concepts. We illustrate the various characteristics of temporal regularization via a sequence of simple discrete and continuous MDPs, and show that the technique provides improvement even in high-dimensional Atari games.
This paper is very interesting. One previous assumption in TD learning is that reward are close with states in proximity of the state space, which has been pointed out by many papers is not realistic and have problems for spatial value function regularization. Instead, this paper make the assumption that rewards are close for states.  Overall this paper has a very good motivation, and the literature review shows that the author is knowledgable of this field. This paper could open a novel area of temporal regularization that received inadequate attention before. Here are my major concerns: 1. The author tries to squeeze too much content to show the effectiveness of the proposed method. For example, the experiments covers both value function approximation and control. To me, the effect of control by the improved value function approximation is indirect and less clear than showing results of policy evaluation. I would suggest the author to remove (or put it to the appendix) the control part (such as line 137-140, and section 5.6) and focus more on the value approximation part. 2. The comparison study of different temporal regularization methods is incomplete. As the author pointed out in line 128, different temporal regularization methods could be used. Then the comparison study of different methods are completely missing in the paper, which makes the paper incomplete from this perspective. 3 connections with previous work: This temporal regularization will serve a very similar purpose as Residual gradient (RG) vs. TD on backward bootstrapping, where the value is affected by both the previous state and the next state. The benefit is stability and the disadvantage of deteriorated prediction. This has been investigated in several previous papers, such as  Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., & Wiewiora, E. (2009, June). Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 993-1000). ACM. (detailed discussed backward bootstrapping)  Li, L. (2008, July). A worst-case comparison between temporal difference and residual gradient with linear function approximation. In Proceedings of the 25th international conference on machine learning (pp. 560-567). ACM. (compare the prediction and stability of TD and RG, though not explicitly bridging the connections with backward bootstrapping)   Some minor suggestions: line 56-67: it is much better to give reasons and toy examples why reversal Markov chain property (Def 2) is less violated in practice than Detailed balance (Def. 1). line 110& 122. It is better to change ‘Property 1’ —> ‘Proposition 1’. line 124 Does this imply the average reward \rho is more robust than the value function V? what else does it imply here? line 135 reference for ’trend-ajusted model’ is missing. === I have read the response, and have updated my reviews.