Maximum Causal Tsallis Entropy Imitation Learning
In this paper, we propose a novel maximum causal Tsallis entropy (MCTE) framework for imitation learning which can efficiently learn a sparse multi-modal policy distribution from demonstrations. We provide the full mathematical analysis of the proposed framework. First, the optimal solution of an MCTE problem is shown to be a sparsemax distribution, whose supporting set can be adjusted. The proposed method has advantages over a softmax distribution in that it can exclude unnecessary actions by assigning zero probability. Second, we prove that an MCTE problem is equivalent to robust Bayes estimation in the sense of the Brier score. Third, we propose a maximum causal Tsallis entropy imitation learning (MCTEIL) algorithm with a sparse mixture density network (sparse MDN) by modeling mixture weights using a sparsemax distribution. In particular, we show that the causal Tsallis entropy of an MDN encourages exploration and efficient mixture utilization while Shannon entropy is less effective.
************************* Summary: ************************* This work introduces a new variant of imitation learning, MCTEIL, that replaces the common approach of maximizing the causal entropy within recent imitation learning approaches with maximizing the causal Tsallis entropy. This allows learning unimodal and multi-modal policies from demonstrations. The work is trained in an adversarial way, similar to GAIL [1], and is compared to behavior cloning, GAIL, and a baseline that is introduced in the paper, soft GAIL.  ************************* Quality: ************************* The approach appears sound, although I did not check the math too heavily. The approach is well-supported with several theoretical proofs. There is no discussion about the weaknesses of the approach.   ************************* Clarity: ************************* The paper is well-written and there were few typos.   One disadvantage is that there were no hyper-parameters or architectural details included for the experiments, and so it is unlikely that the results can be reproduced. However, it is useful that the pseudo-code was provided.   ************************* Originality: ************************* The main contribution to this work is that it describes how to incorporate causal Tsallis entropy into GAIL. This seems like an interesting formulation, however, it is unclear because this paper does not have a related works section. There are a few relevant works mentioned throughout the paper, but they either 1) are directly relevant to describing the background or 2) only mentioned briefly to state why they are not used as baselines. As such, it is unclear how much this work differs from other papers. For example, there are a couple works that were published in NIPS last year, InfoGAIL [2], and Multi-modal imitation learning [3], that learn multi-modal behaviors from demonstrations. The paper states that there are no comparisons with InfoGAIL because it uses a single Gaussian for the policy. However, InfoGAIL uses latent codes to capture multiple modalities. Additionally, the paper claims the latter work was excluded because it focuses on multi-task settings, but the experiments in that paper include both single and multi-task results. As such, it appears that these works should have been included in the comparisons, or at least warranted further discussions.  ************************* Significance: ************************* It is useful to learn multi-modal policies, in particular because they provide alternatives to difficult policies to achieve, and additionally can highlight different preferences among experts. This paper has nice theoretical foundations, and the pseudo-code is pretty clear, so it would be easy to make a couple of adjustments to GAIL if researchers are interested in using MCTEIL. However, because this paper does not compare against relevant works, it is difficult to assess how significant it is.   Additionally, the paper needs more analysis of the results. Why does MCTEIL outperform in Half-Cheetah and Reacher, but not the other environments? Furthermore, there were only 3 averaged runs, so it is unclear how significant the results are.   ************************* Typos ************************* Line 39: modeling expert’s -> modeling the expert’s  Line 109: to the expected -> with the expected  ************************* References: ************************* [1] Ho, Jonathan, and Stefano Ermon. "Generative adversarial imitation learning." Advances in Neural Information Processing Systems. 2016.  [2] Li, Yunzhu, Jiaming Song, and Stefano Ermon. "InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations." arXiv preprint arXiv:1703.08840 (2017).  [3] Hausman, Karol, et al. "Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets." Advances in Neural Information Processing Systems. 2017.  Update: I have read through the reviews and the author's response.  I still think it is necessary to have discussions in the paper relating the work to other relevant literature, but I am willing to change my review to weak accept, given the theoretical contributions and commitment to include the relevant works and experiments in the paper.