Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with a single function. In reinforcement learning, ensemble algorithms such as an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration. In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal difference errors. The advantage of this algorithm is that it improves ensemble performance by reducing weights of Q-functions unfamiliar with current trajectories. We provide experimental results for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms.
This paper addresses the problem of combining separately learned action-value functions into a single, no-longer-learning, action selection algorithm. The result is not necessarily a combined policy, strictly speaking, because it need not take the same action (or produce the same action selection probabilities) each time it is in the same state. Such “ensemble methods” have been a big topic in supervised learning, where ensemble methods are often effective, but less so in reinforcement learning, where the evidence in support of the idea is, perhaps, weak. This paper just assumes it is a good idea, while citing prior work that i don’t think really shows that it is. A new method is proposed, called TDW, based on weighting the learner’s action values according to their recent squared temporal-difference errors. This is compared with simple ensemble method that have no memory other than their action values, on two gridworld problems and six Atari games. Comparison is also made with single (non-ensemble) methods that are given as much training data as one of the members of the ensemble. The results are consistent with TDW being better than the simpler ensemble methods on both gridworlds and Atari games, and with both ensemble methods being better than the single methods on Atari games but worse than the single methods on the gridworld problems. 
Does this work make a contribution to the field of reinforcement learning? A lot depends on whether you think the problem of combining separate learners is an important problem. I am skeptical. First, this problem requires one to be able to identify separate learners that are operating in independent copies of the identical environment, and whose states and actions can be mapped one-to-one onto each another. (This is also a requirement of asynchronous methods, so this criticism applies to them as well.) Of course it is easy for us to arrange these things with our simulated test environments, and there may even be _a few_ real problems like this, but most of the time this is unrealistic. If a method requires this, then it is not a general method. Second, and I suppose relatedly, this new problem bears little relation to the original problem: Does our field benefit from the introduction of new problems and solution methods that apply only in special cases? Or are these special cases primarily distractions from the general case and thus ways to avoid coming to grips with the real problem?
Different researchers might answer these rhetorical questions in different ways. It is a judgement call, but they reflect a real concern. A greater problem is that this work does not do a good job addressing the problem of ensembles. First, the introduction glosses over the question of whether ensembles are a good idea; it kinda suggests that their effectiveness has already been established, but does not commit itself to that statement; it does not make a clear claim that might be true or false about that prior work and that would validly support interest in this area. Second, this paper contains the wrong sort of comparisons of ensemble methods with single (non-ensemble) methods. It compares an ensemble of 10 learners with a single learner who only has as much data as one of the 10 learners in the ensemble. Thus, each single learner has only one-tenth as much data as has gone into the ensemble. Surely this is not the right comparison. It is a real problem that an early proponent of combining learners makes an unfair comparison with non-combined learners. It sets a poor standard and example for those who come later. The third way in which this work on ensemble RL agents is not very good it that its ensemble methods are not well suited to the task. That is, if one was going to combine learners, then there are many interesting natural ways to do this, and these are not done. It would be natural for each learner to keep track of how much experience or confidence it has in each part of the state space. The TDW algorithm can be seen as one way of doing this, but it is just one rather idiosyncratic way; there are many more natural ways. If we are going to explore this new problem, and new algorithms for solving it, then one really ought to do better than propose only methods that retain only their action-value function and nothing else that might facilitate the subsequent combination. In these three ways I feel this paper is not a good example of exploring the new problem of ensemble learning, even if you think that problem is a worthy one.