PARAMETER SPACE NOISE FOR EXPLORATION
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off-and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks.
This paper proposes a method for parameter space noise in exploration.
Rather than the "baseline" epsilon-greedy (that sometimes takes a single action at random)... this paper presents an method for perturbations to the policy.
In some domains this can be a much better approach and this is supported by experimentation.
There are several things to like about the paper:
- Efficient exploration is a big problem for deep reinforcement learning (epsilon-greedy or Boltzmann is the de-facto baseline) and there are clearly some examples where this approach does much better.
- The noise-scaling approach is (to my knowledge) novel, good and in my view the most valuable part of the paper.
- This is clearly a very practical and extensible idea... the authors present good results on a whole suite of tasks.
- The paper is clear and well written, it has a narrative and the plots/experiments tend to back this up.
- I like the algorithm, it's pretty simple/clean and there's something obviously *right* about it (in SOME circumstances).
However, there are also a few things to be cautious of... and some of them serious:
- At many points in the paper the claims are quite overstated. Parameter noise on the policy won't necessarily get you efficient exploration... and in some cases it can even be *worse* than epsilon-greedy... if you just read this paper you might think that this was a truly general "statistically efficient" method for exploration (in the style of UCRL or even E^3/Rmax etc).
- For instance, the example in 4.2 only works because the optimal solution is to go "right" in every timestep... if you had the network parameterized in a different way (or the actions left/right were relabelled) then this parameter noise approach would *not* work... By contrast, methods such as UCRL/PSRL and RLSVI https://arxiv.org/abs/1402.0635 *are* able to learn polynomially in this type of environment. I think the claim/motivation for this example in the bootstrapped DQN paper is more along the lines of "deep exploration" and you should be clear that your parameter noise does *not* address this issue.
- That said I think that the example in 4.2 is *great* to include... you just need to be more upfront about how/why it works and  what you are banking on with the parameter-space exploration. Essentially you perform a local exploration rule in parameter space... and sometimes this is great - but you should be careful to distinguish this type of method from other approaches. This must be mentioned in section 4.2 "does parameter space noise explore efficiently" because the answer you seem to imply is "yes" ... when the answer is clearly NOT IN GENERAL... but it can still be good sometimes ;D
- The demarcation of "RL" and "evolutionary strategies" suggests a pretty poor understanding of the literature and associated concepts. I can't really support the conclusion "RL with parameter noise exploration learns more efficiently than both RL and evolutionary strategies individually". This sort of sentence is clearly wrong and for many separate reasons:
    - Parameter noise exploration is not a separate/new thing from RL... it's even been around for ages! It feels like you are talking about DQN/A3C/(whatever algorithm got good scores in Atari last year) as "RL" and that's just really not a good way to think about it.
    - Parameter noise exploration can be *extremely* bad relative to efficient exploration methods (see section 2.4.3 https://searchworks.stanford.edu/view/11891201)
Overall, I like the paper, I like the algorithm and I think it is a valuable contribution.
I think the value in this paper comes from a practical/simple way to do policy randomization in deep RL.
In some (maybe even many of the ones you actually care about) settings this can be a really great approach, especially when compared to epsilon-greedy.
However, I hope that you address some of the concerns I have raised in this review.
You shouldn't claim such a universal revolution to exploration / RL / evolution because I don't think that it's correct.
Further, I don't think that clarifying that this method is *not* universal/general really hurts the paper... you could just add a section in 4.2 pointing out that the "chain" example wouldn't work if you needed to do different actions at each timestep (this algorithm does *not* perform "deep exploration").
I vote accept.