Learning Plannable Representations with Causal InfoGAN
In recent years, deep generative models have been shown to 'imagine' convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans -a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation 3 .
1. Summary - This paper proposes Causal InfoGAN which is capable of generating a realistic sequence of observations from a given initial observation to a goal observation. The motivation is that this is solving a relaxed version of planning problem, which requires generating a sequence of actions. The idea of Causal InfoGAN is to train a generator that generates two consecutive observations given two consecutive “abstract states”, which encourages the model to generate realistic transitions. In order to enforce the consistency between abstract states and observations, this paper adds a mutual information maximization objective (which is the idea from InfoGAN). The experimental result on a toy 2D domain shows that the proposed model can learn reasonable abstract states that are consistent with the dynamics of the environment, as opposed to simple baselines (e.g., K-means) which are independent of the dynamics of the environment. The qualitative result on Rope Manipulation data shows that Causal InfoGAN can generate more realistic transitions than InfoGAN.   [Pros] - Novel application of GAN that is capable of generating a sequence of observations that is consistent with the dynamics of the environment. - The proposed idea is sensible and interesting.   [Cons] - The experimental result is okay but not much comprehensive.  2. Quality - The idea of training a GAN to generate realistic transitions that is interesting.  - The experimental result could be more comprehensive. First, the paper only presents qualitative results for Rope Manipulation task. Though Casual InfoGAN looks better than InfoGAN, some transitions seem very unrealistic. It would be better to provide quantitative results (something like Table 1). Second, it would be much convincing/interesting to show results on more complex and popular domains such as Atari and MuJoCo. Lastly, it would be good to show an evidence that the learned model can generalize to unseen observations, which would be one of the advantages of model-based RL. - It would be good to discuss the following work in the related work section: Universal planning network [Srinivas et al.] and Model-Based Planning with Discrete and Continuous Actions [Henaff et al.]. They seek to generate a sequence of “actions” given initial/goal observations.  3. Clarity - The problem formulation and the description of the method are well-written. - In Section 2, the paper claims that the planning problem (generating a sequence of actions) is “unnecessarily” difficult to solve, in order to justify their relaxed problem (generating observations). This sounds like a too strong argument. How can we build a control agent if we do not care about generating actions and only generate “observations”?   4. Originality - The idea of learning an abstract transition model using GAN is novel to my knowledge.  5. Significance - I would personally view this work as an interesting application of GAN for dynamical systems rather than “learning plannable representation” that sounds quite ambitious, because the authors are not tackling the full planning problem (e.g., dealing with actions, exploration, reward). Nevertheless, I think this paper is tackling an interesting problem with a new idea, which can be potentially useful for control in the future.