INFORMATION ASYMMETRY IN KL-REGULARIZED RL
Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviours that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.
-- Originality --
This paper studies how to use KL-regularization with information asymmetry to speed up and improve reinforcement learning (RL). Compared with existing work, the major novelty in the proposed algorithm is that it uses a default policy learned from data, rather than a fixed default policy. Moreover, the proposed algorithm also limits the amount of information the default policy receives, i.e., there is an "information asymmetry" between the agent policy and the default policy. In many applications, the default policy is purposely chosen to be "goal agnostic" and hence conducts the "transfer learning". To the best of my knowledge, this "informationally asymmetric" KL-regularization approach is novel.
-- Clarify --
The paper is well written in general and is easy to follow.
-- Significance --
I think the idea of regularizing RL via an informationally asymmetric default policy is interesting. It might be an efficient way to do transfer learning (generalization) in some RL applications. This paper has also done extensive and rigorous experiments. Some experiment results are thought-provoking.
-- Pros and Cons
Pros:
1)  The idea of regularizing RL via an informationally asymmetric default policy is interesting. To the best of my knowledge, this "informationally asymmetric" KL-regularization approach is novel.
2) The experiment results are extensive, rigorous, and thought-provoking.
Cons:
1) My understanding is that this "informationally asymmetric" KL-regularization approach is a general approach and can be combined with many policy learning algorithms. It is not completely clear to me why the authors choose to combine it with an actor-critic approach (see Algorithm 1)? Why not combine it with other policy learning algorithms? Please explain.
2) This paper does not have any theoretical results. I fully understand that it is highly non-trivial or even impossible to analyze the proposed algorithm in the general case. However, I recommend the authors to analyze (possibly a variant of) the proposed algorithm in a simplified setting (e.g. the network has only one layer, or even is linear) to further strengthen the results.
3) The experiment results of this paper are interesting, but I think the authors can do a better job of intuitively explaining the experiment results. For instance, the experiment results show that when the reward is "dense shaping", the proposed method and the baseline perform similarly. Might the authors provide an intuitive explanation for this observation? I recommend the authors to try to provide intuitive explanation for all such interesting observations in the paper.