Distral: Robust Multitask Reinforcement Learning
Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tasks. In practice, however, this is not usually observed, because gradients from different tasks can interfere negatively, making learning unstable and sometimes even less data efficient. Another issue is the different reward schemes between tasks, which can easily lead to one task dominating the learning of a shared model. We propose a new approach for joint training of multiple tasks, which we refer to as Distral (distill & transfer learning). Instead of sharing parameters between the different workers, we propose to share a "distilled" policy that captures common behaviour across tasks. Each worker is trained to solve its own task while constrained to stay close to the shared policy, while the shared policy is trained by distillation to be the centroid of all task policies. Both aspects of the learning process are derived by optimizing a joint objective function. We show that our approach supports efficient transfer on complex 3D environments, outperforming several related methods. Moreover, the proposed learning process is more robust to hyperparameter settings and more stable-attributes that are critical in deep reinforcement learning.
The paper presents an approach to performing transfer between multiple reinforcement learning tasks by regularizing the policies of different tasks towards a central policy, and also encouraging exploration in these policies.  The approach relies on KL-divergence regularization.  The idea is straightforward and well explained.  There are no theoretical results regarding the learning speed or quality of the policies obtained (though these are soft, so clearly there would be some performance loss compared to optimal).  The evaluation shows slightly better results that A3C baselines in both some simple mazes and deep net learning tasks.
While the paper is well written, and the results are generally positive, the performance improvements are modest.  It would have been nice also to quantify the performance in *runtime* instead of just in terms of number of samples used (since the networks architectures are different).  In the tabular case, it would really have been nice to show some more detailed experiments to understand the type of behaviour obtained.  Eg. what happens if goals are sampled uniformly (as in these experiments) vs all sampled in one or the other of the extreme rooms, and what happens as the number of sampled tasks increases. Also, another baseline which does not seem to be included but would be very useful is to simply regularize task policies towards literally a "centroid" of the suggested actions, without learning a distilled policy.  One could simply average the policies of the different tasks and regularize towards that quantity.  It is not clear if this would be competitive with the proposed approach, but conceptually it should lead to very similar results.  If the number of tasks increases, one would expect the distilled policy would in fact simply become uniform - is this the case?
Finally, It would really have been nice to see some attempt at theory in the tabular case. It seems to me that one ought to be able to bound the loss of this approach wrt optimal policies using a variational approach, since the optimization seems to suggest that the algorithm computes a variational approximation to what would be the optimal policy for a distribution of MDPs. Such a result would make the paper more interesting conceptually and give some confidence in its applicability beyond the domains used in the experiments.
Smaller suggestions:
- The title refers to "robustness" which in RL is a technical term regarding optimization wrt an unknown distribution of dynamics (using typically min-max algorithms). This is not the type of approach proposed here, so some clarification is needed.