Reinforcement Learning with Multiple Experts: A Bayesian Model Combination Approach
Potential based reward shaping is a powerful technique for accelerating convergence of reinforcement learning algorithms. Typically, such information includes an estimate of the optimal value function and is often provided by a human expert or other sources of domain knowledge. However, this information is often biased or inaccurate and can mislead many reinforcement learning algorithms. In this paper, we apply Bayesian Model Combination with multiple experts in a way that learns to trust a good combination of experts as training progresses. This approach is both computationally efficient and general, and is shown numerically to improve convergence across discrete and continuous domains and different reinforcement learning algorithms.
UPDATE: I have read the authors' response, and it was particularly helpful to me in showing that I misinterpreted some of the reward modeling.  I now understand better that the learning from experts is actually learning potentials, and yes in that case there is a theoretical convergence guarantee, so those parts of my criticism have been addressed.  I do feel there is still an experimental gap here though, because no other approaches from the Bayesian RL or other literature were tested against,a and the number of variants of the current algorithm tested are relatively small and in toy domains.  Summary: The paper describes a Bayesian method for combining the potential functions of multiple experts over time in a reinforcement learning scenario.  A method is given for updating both the posterior probabilities of the potentials and calculating the weighted reward function used in choosing an action.  Empirical results in a gridworld and cart pole are given.  Review:  I think the problem setting the authors have chosen is very rich and poses an interesting problem for a reinforcement learning agent.  I also like that the expert advice is provided in the form of potentials, rather than trajectories or exact reward functions, because it makes the objective clearly defined (maximize the agent’s own reward) and the role of the experts (to speed up exploration) clear as well.  However, I thought the solution wasn’t quite worked out or tested to the level of detail needed for a NIPS paper and comparisons to other Bayesian approaches were not attempted.  Specifically, the paper could be substantially improved based on the following:  The algorithm itself is Bayesian in that given the data already collected, it calculates a posterior on the potential expert potentials.  But the algorithm does not really make use of that knowledge in its action selection.  That is, the weighted potentials do certainly affect exploration, but the algorithm does not take actions to explore *which* expert might be right, which is a hallmark of Bayesian RL algorithms.    A good, and very relevant, example of such an algorithm is the BOSS algorithm from “A Bayesian Sampling Approach to Exploration in Reinforcement Learning” (UAI 2009).  That algorithm has a space of potential models, and performs exploration in such a way that either explicitly refutes some models or follows an optimal strategy.  One could perform similar exploration of the potentials in order to better explore the weight space, and I was surprised to not see a Bayesian RL approach here that did so.  It seems like a comparison to a Bayesian RL algorithm that treats the experts as potential models is needed.  In addition, the empirical results did not cover enough scenarios to be completely convincing.  The current results definitely show the mixed potentials are quickly converging to the performance of a good potential function.  But what if the mixture did not contain a good potential?  Or contained 9 bad functions and 1 good one?  How does the performance of the algorithm vary as that good/bad mixture varies?   If this algorithm is going to be used in larger environments, there will likely be many more experts and potentials, and we need to know how the algorithm will behave under these conditions.  From a theoretical perspective, the paper lacks a rigorous theoretical result proving convergence or ensuring optimal asymptotic behavior.  Without such guarantees, and only the limited experiments, the algorithm does not feel like it has been proven enough.   I also found the result at the end of Section 3.1 confusing.  Doesn’t the result that the true rewards can be found by a weighted combination of the experts require that the value function be in the convex hull of their individual functions?  Why is that assumption not stated outright?  Again, it is unclear how the algorithm will behave with all very incorrect potentials.  Minor points:  The paper makes a number of very general claims about how it can be applied with any exploration policy, any RL algorithm, etc., but then goes on to state there are no theoretical guarantees and provides an implementation algorithm that may only be applicable for model free learning.  Would Rmax work like this?  I think the general claims may be a little overreaching, though I think a large class of algorithms are indeed covered.  Lines 142-145 make reference to equation numbers that do not appear above  The derivation in lines 141-142 is somewhat hard to follow and could be improved by including a numeric example of the computation.