DIRECTED-INFO GAIL: LEARNING HIERARCHICAL POLICIES FROM UNSEGMENTED DEMONSTRATIONS USING DIRECTED INFORMATION
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting stateaction trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.
The paper presents a learning-based method for learning the latent context codes from demonstrations along with a GAIL model. 
This amounts to learning the option segments and the policies simultaneously. 
The main contribution is the model the problem as a time-dependent context and then use a directed information flow loss instead of the mutual information loss.
1. What is the effect of models of the underlying distribution of latent codes. 
Can it be categorical only, or can it be continuous? 
Could we also model it as multidimensional?
The current results only provide single dimensional categorial distribution as latent codes. 
2. The paper missed an important line of work which solves nearly the same problem -- option discovery and policy learning. 
Krishnan -- Discovery of Deep Option(1703.08294). This work was used by authors in continuous options and then again for program generation (https://openreview.net/pdf?id=rJl63fZRb). 
They explicitly infer the option parameters, along with termination conditions with the Expectation Propagation method. 
The results are in very similar domains hence comments, if not a comparison, would be useful. 
3. The authors state that the main problem with an InfoGail style method is dependence on the full trajectory as in eq 1. Hence the directed info flow is required to solve the problem. However in the actual model, the authors make a sequence of variational approximations -- (a) reduction of eq2 to eq1 with a variation lower bound on posterior p(c|c,\tau) and then replace the prior p(c) with q(c|c,\tau) in eq 5. But looking at the model diagram in fig 2. the VAE actually makes the Markovian assumption -- i.e. c only depends on c_{t-1} and s_{t}. If that is true then how would this be very different from InfoGAIL mutual information loss. 
It appears that to capture the authors' mathematical intuition the VAE should have a recurrent generator which should have a hidden state factor passing in to capture dependence on history until the current time. 
3a. In fact the first term in eq 6 looks closer to the actually used model. If that is not true then the authors should clarify. 
4. Experiments do capture the notion discovery of options. But the simplicity of data leaves much to be desired. 
One of the main difference of this work in comparison to unsupervised segmentation models GMM or BP-AR-HMM is the fact that the options learned are composable. But the authors only show this composability on the circle domain -- which is arguably a toy-domain. 
A reasonable confirmation that the model indeed learns composition is to generate a trajectory for a sequence of latent code not seen in data. -- like walking -- normal -- left-right-left can be converted to limping gait -- left-left-right-right. This is only a suggestive example. 
5. In appendix eq 8 how is the reduction from line 3 to line 4 of the equation made -- what is the implicit assumption. 
joint distribution p(c, \tau) is written out as p (\tau|c) p(c) without an integral.