Published as a conference paper at ICLR 2018 NOISY NETWORKS FOR EXPLORATION
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and Dueling agents (entropy reward and -greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.
This paper introdues NoisyNets, that are neural networks whose parameters are perturbed by a parametric noise function, and they apply them to 3 state-of-the-art deep reinforcement learning algorithms: DQN, Dueling networks and A3C. They obtain a substantial performance improvement over the baseline algorithms, without explaining clearly why.
The general concept is nice, the paper is well written and the experiments are convincing, so to me this paper should be accepted, despite a weak analysis.
Below are my comments for the authors.
---------------------------------
General, conceptual comments:
The second paragraph of the intro is rather nice, but it might be updated with recent work about exploration in RL.
Note that more than 30 papers are submitted to ICLR 2018 mentionning this topic, and many things have happened since this paper was
posted on arxiv (see the "official comments" too).
p2: "our NoisyNet approach requires only one extra parameter per weight" Parameters in a NN are mostly weights and biases, so from this sentence
one may understand that you close-to-double the number of parameters, which is not so few! If this is not what you mean, you should reformulate...
p2: "Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent."
Two ideas seem to be collapsed here: the idea of diminishing noise over an experiment, exploring first and exploiting later, and the idea of
adapting the amount of noise to a specific problem. It should be made clearer whether NoisyNet can address both issues and whether other
algorithms do so too...
In particular, an algorithm may adapt noise along an experiment or from an experiment to the next.
From Fig.3, one can see that having the same initial noise in all environments is not a good idea, so the second mechanism may help much.
BTW, the short section in Appendix B about initialization of noisy networks should be moved into the main text.
p4: the presentation of NoisyNets is not so easy to follow and could be clarified in several respects:
- a picture could be given to better explain the structure of parameters, particularly in the case of factorised (factorized, factored?) Gaussian noise.
- I would start with the paragraph "Considering a linear layer [...] below)" and only after this I would introduce \theta and \xi as a more synthetic notation.
Later in the paper, you then have to state "...are now noted \xi" several times, which I found rather clumsy.
p5: Why do you use option (b) for DQN and Dueling and option (a) for A3C? The reason why (if any) should be made clear from the clearer presentation required above.
By the way, a wild question: if you wanted to use NoisyNets in an actor-critic architecture like DDPG, would you put noise both in the actor and the critic?
The paragraph above Fig3 raises important questions which do not get a satisfactory answer.
Why is it that, in deterministic environments, the network does not converge to a deterministic policy, which should be able to perform better?
Why is it that the adequate level of noise changes depending on the environment? By the way, are we sure that the curves of Fig3 correspond to some progress
in noise tuning (that is, is the level of noise really "better" through time with these curves, or they they show something poorly correlated with the true reasons of success?)?
Finally, I would be glad to see the effect of your technique on algorithms like TRPO and PPO which require a stochastic policy for exploration, and where I believe that the role of the KL divergence bound is mostly to prevent the level of stochasticity from collasping too quickly.
-----------------------------------
Local comments:
The first sentence may make the reader think you only know about 4-5 old works about exploration.
Pp. 1-2 : "the approach differs ... from variational inference. [...] It also differs variational inference..."
If you mean it differs from variational inference in two ways, the paragraph should be reorganized.
p2: "At a high level our algorithm induces a randomised network for exploration, with care exploration
via randomised value functions can be provably-efficient with suitable linear basis (Osband et al., 2014)"
=> I don't understand this sentence at all.
At the top of p3, you may update your list with PPO and ACKTR, which are now "classical" baselines too.
Appendices A1 and A2 are a lot redundant with the main text (some sentences and equations are just copy-pasted), this should be improved.
The best would be to need to reject nothing to the Appendix.
---------------------------------------
Typos, language issues:
p2
the idea ... the optimization process have been => has
p2
Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent.
=> you should make a sentence...
p3
the the double-DQN
several times, an equation is cut over two lines, a line finishing with "=", which is inelegant
You should deal better with appendices: Every "Sec. Ax/By/Cz" should be replaced by "Appendix Ax/By/Cz".
Besides, the big table and the list of performances figures should themselves be put in two additional appendices
and you should refer to them as Appendix D or E rather than "the Appendix".