NADPEX: AN ON-POLICY TEMPORALLY CONSISTENT EXPLORATION METHOD FOR DEEP REINFORCEMENT LEARNING
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for tasks with either sparse rewards or long term information. In this work, we introduce a novel on-policy temporally consistent exploration strategy -Neural Adaptive Dropout Policy Exploration (NADPEx) -for deep reinforcement learning agents. Modeled as a global random variable for conditional distribution, dropout is incorporated to reinforcement learning policies, equipping them with inherent temporal consistency, even when the reward signals are sparse. Two factors, gradients' alignment with the objective and KL constraint in policy space, are discussed to guarantee NADPEx policy's stable improvement. Our experiments demonstrate that NADPEx solves tasks with sparse reward while naive exploration and parameter noise fail. It yields as well or even faster convergence in the standard mujoco benchmark for continuous control.
The authors propose a new on-policy exploration strategy by using a policy with a hierarchy of stochasticity. The authors use a two-level hierarchical distribution as a policy, where the global variable is used for dropout. This work is interesting since the authors use dropout for policy learning and exploration.  The authors show that parameter noise exploration is a particular case of the proposed policy. The main concern is the gap between the problem formulation and the actual optimization problem in Eq 12. I am very happy to give a higher rating if the authors address the following points. 
Detailed Comments 
(1) The authors give the derivation for Eq 10. However, it is not obvious that how to move from line 3 to line 4 at Eq 15.
Minor:  Since the action is denoted by "a",  it will be more clear if the authors use another symbol to denote the parameter of q(z) instead of "\alpha" at Eq 10 and 15.
(2) Due to the use of the likelihood ratio trick, the authors use the mean policy as an approximation at Eq 12. Does such approximation guarantee the policy improvement? Any justification?
(3) Instead of using the mean policy approximation in Eq 12, the authors should consider existing Monte Carlo techniques to reduce the variance of the gradient estimation. For example, [1] could be used to reduce the variance of gradient w.r.t. \phi. Note that the gradient is biased if the mean policy approximation is used.
(4) Are \theta and \phi jointly and simultaneously optimized at Eq 12?  The authors should clarify this point. 
(5) Due to the mean policy approximation, does the mean policy depend on \phi? The authors should clearly explain how to update \phi when optimizing Eq 12. 
(6) If the authors jointly and simultaneously optimize \theta and \phi, why a regularization term about q_{\phi}(z)  is missing in Eq 12 while a regularization term about \pi_{\theta|z} does appear in Eq 12? 
(7) The authors give the derivations about \theta such as the gradient and the regularization term about \theta (see, Eq 18-19). However, the derivations about \phi are missing.  For example, how to compute the gradient w.r.t. \phi? Since the mean policy is used, it is not apparent that how to compute the gradient w.r.t. \phi. 
Minor, 1/2 is missing in the last line of Eq 19.
Reference:
[1] AUEB, Michalis Titsias RC, and Miguel Lázaro-Gredilla. "Local expectation gradients for black box variational inference." In Advances in neural information processing systems, pp. 2638-2646. 2015.