Distributed Multitask Reinforcement Learning with Quadratic Convergence
Multitask reinforcement learning (MTRL) suffers from scalability issues when the number of tasks or trajectories grows large. The main reason behind this drawback is the reliance on centeralised solutions. Recent methods exploited the connection between MTRL and general consensus to propose scalable solutions. These methods, however, suffer from two drawbacks. First, they rely on predefined objectives, and, second, exhibit linear convergence guarantees. In this paper, we improve over state-of-the-art by deriving multitask reinforcement learning from a variational inference perspective. We then propose a novel distributed solver for MTRL with quadratic convergence guarantees.
Summary: This paper presents a distributed Multitask Reinforcement Learning (MTRL) from a variational inference perspective. It improves the state-of-art ADMM by a distributed solver using Chebyshev polynomials with \textit{quadratic convergence guarantees}. The theoretical analysis is verified with extensive experiments, but to strengthen the result, further experiments on real large-scale Multitask Reinforcement Learning problems are needed.  Quality: Technically, this paper is rather strong. First, the MTRL is formulated as a variational inference problem by introducing a variational trajectory distribution to arrive at the maximization of variational lower bound. Variational distribution and policy parameters (shared and task-specific) could be optimized alternatively. Furthermore, to learn the shared parameter, the authors incorporate Laplacian matrices from Graph structure into the objective function, with a consensus constraint for the shared parameter to share and transfer knowledge. To solve the scalability issues, the authors adopt Chebyshev polynomials and de-centralize, distribute the computations of Newton direction, and prove the quadratic convergence rate.  The experiments' scales are pretty small, even for the larger random network with only 150 nodes and 250 edges. It is better to study the really large-scale problems and investigate the speedup w.r.t computation units e.g. the number of CPU/GPU.  Clarity: The majority of this paper is clear and well written and organized. However, there are some minor issues. -- Notation inconsistency: for action, line 57, 58, 67 use $a$, while line 103 uses $u$; for $q$, line 208 denotes dual function $q(\lambda)$, yet line 76 denotes variational distribution $q_\phi(\tau)$. -- Figures: Fig. 2(a) is too small, it might be better to use log-log plot for Fig. 6 -- Algorithm: it might be better to move algorithm 1 into the main paper, instead of the Appendix. -- Experiment: As communication overhead is not ignorable, it is better to show the comparison of overall time (communication + running time). -- Computing Architecture: It is better to illustrate how master and worker nodes are organized to distribute the Newton direction computation. This will help us understand the distribution algorithm and communication overhead. -- Results: In Fig. 3(b) for CP, SDD-Newton does not seem to be the best. In Fig. 6(c), distributed SDD Newton is no better than Distributed ADMM for the final objective values.  Originality: The major contribution of this paper is to adopt Chebyshev polynomials and distribute the computing of Newton search direction with a proof of quadratic convergence rate. It is natural to use Chebyshev polynomials to solve the system of linear equations described via diagonally dominant matrices, where the Newton search direction is such kind of problem. However, the distributed Chebyshev Solver is non-trivial and it is an interesting idea to introduce such a technique into distributed Multitask Reinforcement Learning.  Significance: Scalability is an important issue in Multitask Reinforcement Learning and it is the right direction to distribute the computation to as many machines as possible. Thus, the quadratic convergence guarantees an important result for the distributed Multitask Reinforcement Learning. However, the trade-off between communication and computation is the next important issue needed to be seriously considered.  ------- After Rebuttal --------- I have checked the others' reviews. Most of my concerns are addressed by the author feedback.