Imitation learning relies on expert demonstrations. Existing approaches often require that the complete demonstration data, including sequences of actions and states are available. In this paper, we consider a realistic and more difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are not available. Inferring the unseen expert actions in a stochastic environment is challenging and usually infeasible when combined with a large state space. We propose a novel policy learning method which only utilizes the expert state sequences without inferring the unseen actions. Specifically, our agent first learns to extract useful sub-goal information from the state sequences of the expert and then utilizes the extracted sub-goal information to factorize the action value estimate over state-action pairs and subgoals. The extracted sub-goals are also used to synthesize guidance rewards in the policy learning. We evaluate our agent on five Doom tasks. Our empirical results show that the proposed method significantly outperforms the conventional DQN method.
SIGNIFICANCE AND ORIGINALITY:
The authors propose to accelerate the learning of complex tasks by exploiting traces of experts.
Unlike the most common form of imitation learning or behavioral cloning, the authors 
formulate their solution in the case where the expert’s state trajectory is observable, 
but the expert’s actions are not. This is an important and useful problem in robotics and other
applications. Within this specific setting the authors differentiate their approach from others 
by developing a solution that does NOT estimate an explicit dynamics model ( e.g.,  P( S’ | S, A ) ).
The benefits of not estimating an explicit action model are not really demonstrated in a clear way.
The author’s articulate a specific solution that provides heuristic guidance rewards that cause the 
learner to favor actions that achieve subgoals calculated from expert behavior
and refactors the representation of the Q function so that it 
has a component that is a function of the subgoal extracted from the expert.
These subgoals are linear functions of the expert’s change in state (or change in state features).
The resultant policy is a function of the expert traces on which it depends.
The authors show they can retrain a new policy that does not require the expert traces.
As far as I am aware, this is a novel approach to the problem. 
The authors claim that this factorization is important and useful but the paper doesn’t
really illustrate this well.
They demonstrate the usefulness of the algorithm against a DQN baseline on Doom game problems.
The algorithm learns faster than unassisted DQN as shown by learning curve plots. 
They also evaluate the algorithms on the quality of the final policies for their approach, DQN, 
and  a supervised learning from demonstration approach ( LfD ) that requires expert actions.
The proposed approach does as well or better than competing approaches.
QUALITY
Ablation studies show that the guidance rewards are important to achieving the improved performance of the proposed method which is important confirmation that the architecture is working in the intended way. However, it would also be useful to do an ablation study of the “factorization” of action values.  Is this important to achieving better results as well or is the guidance reward enough? This seems like a key claim to establish.
CLARITY
The details of the memory based kernel density estimation and neural gradient training seemed
complicated by the way that the process was implemented. Is it possible to communicate
the intuitions behind what is going on?
 
I was able to work out the intuitions behind the heuristic rewards, but I still don’t clearly get 
what the Q-value factorization is providing:
To keep my text readable, I assume we are working in feature space
instead of state space and use different letters for learner and expert:
   Learner: S = \phi(s) 
   Expert’s i^th state visit:  Ei = \phi( \hat{s}_i }  where Ei’ is the successor state to Ei
The paper builds upon approximate n-step discrete-action Q-learning 
where the Q value for an action is a linear function of the state features:
    Qp(S,a) = Wa S + Ba
where parameters p = ( Wa, Ba ).
After observing an experience ( S,A,R,S’ ) we use Bellman Error as a loss function to optimize Qp for parameter p.
I ignore the complexities of n-step learning and discount factors for clarity.
    Loss = E[    R + MAXa’ Qp(S’,a’)    -   Qp(S,a)   ]  
The authors suggest we can augment the environment reward R 
with a heuristic reward Rh proportional to the similarity between 
the learner “subgoal" and the expert “subgoal" in similar states. 
The authors propose to use cosine distance between representations 
of what they call the “subgoals” of learner and expert. 
A subgoal is defined as a linear transformation of the distance traveled by an agent during a transition.
The heuristic reward is proportional to the cosine distance between the learner and expert “subgoals"
   Rh = B  <   Wv LearnerDirectionInStateS,   
                     Wv ExpectedExpertDirectionInStatesSimilarToS   >
The learner’s direction in state S is just (S-S’) in feature space.
The authors model the behavior of the expert as a kernel density type approximator
giving the expected direction of the expert starting from a states similar to the one the learner is in. 
Let < Wk S, Wk Ej > be a weighted similarity between learner state features S and expert state features Ej
and Ej’ be the successor state features encountered by the expert.
Then the expected expert direction for learner state S is:
     SUMj  < Wk S, Wk Ej > ( Ej - Ej’ ) 
Presumably the linear Wk transform helps us pick out the important dimensions of similarity between S and Ej.
Mapping the learner and expert directions into subgoal space using Wv, the heuristic reward is
   Rh = B <   Wv (S-S’),  
                    Wv SUMj  < Wk S, Wk Ej > ( Ej - Ej’ ) >
I ignore the ReLU here, but I assume that is operates element-wise and just clips negative values?
There is only one layer here so we don’t have complex non-linear things going on?
In addition to introducing a heuristic reward term, the authors propose to alter the Q-function
to be specific to the subgoal.
   Q( s,a,g ) = g(S) Wa S + Ba
The subgoal is the same as the first part, namely a linear transform of the expected expert direction in 
states similar to state S.
    g(S) =  Wv   SUMj  < Wk S, Wk Ej >  ( Ej - Ej’ ) 
So in some sense, the Q function is really just a function of S, as g is calculated from S.
    Q( S,a ) = g(S) Wa S + Ba 
So this allows the Q-function more flexibility to capture each subgoal in a different linear space?
I don’t really get the intuition behind this formulation. It allows the subgoal to adjust the value 
of the underlying model? Essentially the expert defines a new Q-value problem at every state 
for the learner? In some sense are we are defining a model for the action taken by the expert?
ADDITIONAL THOUGHTS
While the authors compare to an unassisted baseline, they don’t compare to methods that use an action model
which is not a fatal flaw but would have been nice. 
One can imagine there might be scenarios where the local guidance rewards of this 
form could be problematic, particularly in scenarios where the expert and learner are not identical
and it is possible to return to previous states, such as the grid worlds the authors discuss:
If the expert’s first few transitions were easily approximable,
the learner would get local rewards that cause it to mimic expert behavior.
However, if the next step in the expert’s path was difficult to approximate, 
then the reward for imitating the expert would be lower.
Would the learner then just prefer to go back towards those states that it can approximate and endlessly loop?
In this case, perhaps expressing heuristic rewards as potentials as described in Ng’s shaping paper might solve the problem.
PROS AND CONS
Important problem generally. Avoiding the estimation of a dynamics model was stated as a given, but perhaps more could be put into motivating this goal. Hopefully it is possible to streamline the methodology section to communicate the intuitions more easily.