Sparse Multi-Task Reinforcement Learning
In multi-task reinforcement learning (MTRL), the objective is to simultaneously learn multiple tasks and exploit their similarity to improve the performance w.r.t. single-task learning. In this paper we investigate the case when all the tasks can be accurately represented in a linear approximation space using the same small subset of the original (large) set of features. This is equivalent to assuming that the weight vectors of the task value functions are jointly sparse, i.e., the set of their non-zero components is small and it is shared across tasks. Building on existing results in multi-task regression, we develop two multi-task extensions of the fitted Q-iteration algorithm. While the first algorithm assumes that the tasks are jointly sparse in the given representation, the second one learns a transformation of the features in the attempt of finding a more sparse representation. For both algorithms we provide a sample complexity analysis and numerical simulations.
Summary: This paper addresses the problem of multi-task reinforcement learning. The underlying intuition is that the value functions of all tasks can jointly be represented as a sparse combination of given basis functions. The paper introduces two algorithms, which are both variants of the Fitted Q-Iteration algorithm. The difference between these two algorithms is how they solve the regression problem at each iteration of FQI.To exploit the joint sparsity, Group LASSO Fitted Q-Iteration (GL-FQI) solves a group Lasso regression problem at each iteration. The regularizer in the Group Lasso encourages finding a sparse set of weights that are jointly used in all tasks.The other algorithm is Feature Learning FQI (FL-FQI). The underlying assumption of FL-FQI is that there exists a linear transformation of the features that make the tasks jointly sparse in the transformed space.With certain set of assumptions, the paper provides error upper bound for both algorithms.One may summarize the main finding of the theoretical guarantee for GL-FQI (FL-FQI) as that the joint sparsity level (the rank of weight matrix) appears in the upper bound. Moreover, if the number of tasks (T) is large, the effect of the number of features (d) becomes less important (the behavior would be log(d) / T for GL-FQI and d / T for FL-FQI). This latter point makes these methods different from a single-task Lasso FQI, which has log(d) behavior (no division by T).The paper also has experimental results comparing LASSO FQI, GL-FQI, and FL-FQI.This is an interesting paper. It is written clearly and has both theory and experiments (though please refer to my comments below regarding the theory, as some aspects of it are not clear). Benefitting from sparsity is not new in RL (both in the API and AVI contexts), but as far as I know, it is new in the multi-task RL setup. Even though GP-FQI might not seem to be a surprisingly novel algorithm, I found FL-FQI quite compelling as it tries to find a transformation that makes the problem sparse.Quality: The paper has both theory and experiments. The theory has some simplifying assumptions (especially Assumptions 1 and 2), but the message is clear: for some problems we might benefit from the joint sparsity of value functions. But there are some concerns about the theory, which I describe below.Clarity: The paper is well-written and well-organized. Some parts of it can be improved though.Originality: The general structure of the algorithms is not surprisingly novel (in particular, GL-FQI), but as far as I know, they have not been suggested before.Significance: The algorithms developed in this paper are good additions to the toolbox of RL practitioners.Detailed Comments (roughly ordered according to their importance):[Major] Please address these comments.* In Appendix C, the authors used a result from Lounici et al. 2011. Is it possible to be a bit more specific about the referred result?If it is Corollary 4.1 in that paper, then their noise model is for Gaussian noise. This is not the case here. Also the constants should be 128/k^2 instead of 1280/k^4.If it is Theorem 8.1, which is for non-Gaussian case, then the bound would be different (we have (log d)^{3/2 + delta} / sqrt{T} ) behavior instead of log d/T behavior).* If I understand Assumption 2 correctly, it indicates that the the i-th basis function phi_i is uncorrelated to (P phi_j). Later Lemma 7 shows that this along Assumption 1 imply that the application of the Bellman operator on a function f does not change its sparsity property. I have two main concerns here:1) Why the probability distribution for this correlation is chosen to be the Lebesgue measure? Why not mu or rho?2) In the proof of Lemma 7, w_1 is the weight that leads to a function f_{w_1} that has zero L_2-norm distance from T f_{w_0}. But this does not imply that they are pointwise equal. If my claim is correct and I did not misunderstood anything, isn't it possible that we have a large difference on a set of measure 0 (w.r.t. the Lebesgue measure)?I believe if we require that the design matrix does not sample from those small sets, the analysis should be fine. Please comment on this issue.* The value of k(s) in Assumption 4 is random, since the design matrix is random. This makes the upper bounds of both theorems random. Also the work of Lounici et al. 2011 is for fixed design. How does it affect the results?* One other issue is that you reuse the dataset in all iteration. It makes them dependent.[Minor] If you have enough space, please address these concerns.* L118:  Assumption 1 is much stronger than requiring that Q* belong to the function space. Is there any specific reason not to study the function approximation error? Is it because Proposition 1 in the supplementary material does not hold?* L095: Features are defined over the state space, but not state-action space.* L213: In the definition of Restricted Eigenvalues, should we have |J| \leq s or |J| \leq \tilde{s} ?* In both Theorems 1 and 2, the probability of the event is written in a non-standard form. Is it possible to simplify it in the usual (eps,delta) form?* What would happen if instead of l_{2,1}-norm on W, we use l_{1,1}? What is the motivation of using l_2-norm of the rows?* A short discussion on how to solve Equation (5)?* A relevant work is recently done by Brunskill and Li, 2013. Please compare with it.* L220: Is the distribution of samples rho or mu? Based on the definition of the concentrability coefficient, it seems that it should be mu.* L287: Typo: "Let X the" ---> "Let X be the"* In Assumption 5, is O^d orthogonal or orthonormal (as suggested on L102)?- Lounici, Pontil, Van De Geer, Tsybakov, et al, "Oracle inequalities and optimal inference under group sparsity," The Annals of Statistics, 2011.- Brunskill and Li, "Sample complexity of multi-task reinforcement learning," UAI, 2013.