We develop a normative theory of hierarchical model-based policy optimization for Markov decision processes resulting in a full-depth, full-width policy iteration algorithm. This method performs policy updates which integrate reward information over all states at all horizons simultaneously thus sequentially maximizing the expected reward obtained per algorithmic iteration. Effectively, policy path programming ascends the expected cumulative reward gradient in the space of policies defined over all state-space paths. An exact formula is derived which finitely parametrizes these path gradients in terms of action preferences. Policy path gradients can be directly computed using an internal model thus obviating the need to sample paths in order to optimize in depth. They are quadratic in successor representation entries and afford natural generalizations to higher-order gradient techniques. In simulations, it is shown that intuitive hierarchical reasoning is emergent within the associated policy optimization dynamics.
This paper presents a reinforcement learning method that exploits the full-depth backup. The policy update based on the full-depth backup is derived for entropy-regularized MDP. The state-action correlation function is introduced and the Fisher information matrix is computed with it. The proposed method is evaluated on tasks with discrete states and actions.
I understand the concept of using the full path for updating the policy, but I do not see significant novelty of the proposed method from the current manuscript. The proposed method looks equivalent to the natural policy gradient with full-depth backup for entropy-regularized MDP, which is a special case of existing methods. 
My concern is that the scalability of the proposed method. The use of the full-depth backup should suffer from the large variance, and I think the proposed method will not work on tasks with the high-dimensional state space. 
The evaluation is limited to tasks of which the state space is small, and the proposed method is not compared with existing methods.
Due to the unclear novelty and limited empirical results, I give weak reject to the paper in the current form.
I request authors to answer the following questions to improve the clarity.
- Is the proposed method equivalent to use the natural policy gradient with the full-depth backup for a softmax energy-based policy? If they are different, what it the crucial difference?
- I think that the variance of the estimation of the gradient is large when using the full-depth back up. I'm curious about the performance of the proposed method in high-dimensional tasks. However, the evaluation is very limited to simple tasks in which the state space is relatively small compared with tasks commonly used in deep RL papers.
Does the proposed method scale to more complex tasks, such as Atari games?
- When using the n-step TD learning, increasing n does not always improve the performance, and n should be set to an intermediate value
What is the motivation of using the full paths for updating the policy? Does the proposed method outperform existing methods? Especially, the comparison with natural policy gradient methods is necessary to show the benefit of the proposed algorithm.
Minor comments:
- In page 2, "A state-space \mathcal{X} is composed of states x \in \mathcal{X}" <- Authors may want to replace x with s in this sentence.
 - In page 6, "0\lambda < 1" I think that "<" is missing between "0" and "\lambda"
- In page 6, some variables are explained after Equation (10). However, it took me a while to find "lambda" in Equation (10), since Equation (10) has 9 lines and many terms. I think the description in page 6 can be improved. For example, I recommend to use "\exp" instead of "e^" for readability.