Reinforcement learning agents learn by exploring the environment and then exploiting what they have learned. This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state. The cost of this freedom is reinforcement learning is slower and more unstable than supervised learning. We explore the possibility that ensemble methods can remedy these shortcomings and do so by investigating a novel technique which harnesses the wisdom of the crowds by bagging Q-function approximator estimates. Our results show that this proposed approach improves all three tasks and reinforcement learning approaches attempted. We are able to demonstrate that this is a direct result of the increased stability of the action portion of the state-actionvalue function used by Q-learning to select actions and by policy gradient methods to train the policy.
This paper proposes the deep reinforcement learning with ensembles of Q-functions. Its main idea is updating multiple Q-functions, instead of one, with independently sampled experience replay memory, then take the action selected by the ensemble. Experimental results demonstrate that the proposed method can achieve better performance than non-ensemble one under the same training steps, and the decision space can also be stabilized.
This paper is well-written. The main ideas and claims are clearly expressed. Using ensembles of Q-function can naturally reduce the variance of decisions, so it can speed up the training procedure for certain tasks. This idea is simple and works well. The main contribution is it provides a way to reduce the number of interactions with the environment. My main concern about the paper is the time cost. Since the method requires updating multiple Q-functions, it may cost much more time for each RL time step, so I’m not sure whether the ensemble method can outperform the non-ensemble one within the same time period. This problem is important for practical usage. However, the authors didn’t show these results in the paper.
Minor things:
+The main idea is described too sketchily. I think more examples, such as in section 8.1, should be put in the main text.
+Page6 Line2, duplicated ‘the’.