Learning Task Specifications from Demonstrations
Real-world applications often naturally decompose into several sub-tasks. In many settings (e.g., robotics) demonstrations provide a natural way to specify the sub-tasks. However, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the sub-tasks can be safely recombined or limit the types of composition available. Motivated by this deficit, we consider the problem of inferring Boolean non-Markovian rewards (also known as logical trace properties or specifications) from demonstrations provided by an agent operating in an uncertain, stochastic environment. Crucially, specifications admit well-defined composition rules that are typically easy to interpret. In this paper, we formulate the specification inference task as a maximum a posteriori (MAP) probability inference problem, apply the principle of maximum entropy to derive an analytic demonstration likelihood model and give an efficient approach to search for the most likely specification in a large candidate pool of specifications. In our experiments, we demonstrate how learning specifications can help avoid common problems that often arise due to ad-hoc reward composition.
** update ** I increase the score I attribute to this paper since the authors' response to the various problems raised by the reviewers seems to me very satisfactory. However the authors promise several improvements and my score cannot be based solely on promises, so this score only increases by one. Finally, the practical motivations of this little explored field are extremely justified, so I also encourage authors to apply their work to more complex problems, which should lead to interesting results.  ** summary ** This paper sets up an algorithm capable from demonstrations (sequence of states and actions)  and from a set of specifications (set of trajectories), to infer the specifications  governing the actions of the agent and the visited states. These specifications can be seen as non-Markovian reward functions.  Thus, this work is related to inverse reinforcement learning (IRL)  which aims to infer the reward function of an agent  by observing these successive states and actions. By defining the probability of a trajectory knowing a specification  (using the maximum entropy principle) the development leads to a posterior distribution. Two algorithms result from this and allow to test the approach  on the system presented in introduction (motivating the paper).  ** quality ** Although its introduction is interesting, this paper is quite hard to follow,  especially from page 4.  The steps should be better justified or motivated by intuition or pedagogical explanations  of the ongoing development. Also some proofs only appears in the supplementary materials and not in the paper,  making the paper less self-contained. At least sketches of proof should be provided to let the reader study the main arguments.  ** clarity ** As mentioned above, clarity is not the strength of this paper.  This can also be illustrated by the more or less in-depth comments that follow.   page 3:  < Sec > Section  The Markov property is already present in Definition 2.1.  < no \phi =^{def} 1 - \phi(\xi) > no \phi(\xi) =^{def} 1 - \phi(\xi)  page 4:  In Equation 2, it seems that we multiply a probability of \xi  (trajectory probability given by the transition functions)  with another probability of \xi (given by the exponential function).  Some confusion in the notations X and \xi.  page 6: < the following simple algorithm. > the following simple algorithm (Algorithm 2).  page 8: < queries,\phi^{hat} > queries, \phi^{hat}  < assignments.\footnote{...} > assignments\footnote{...}.  The conclusion should be a Section.  ** originality ** To the best of my knowledge this paper is the first research  on inferring specifications from demonstrations in the MDP framework. The authors have done a good bibliographical work to position their study  in the relevant literature. I suggest authors read  (Drougard "Exploiting Imprecise Information Sources in Sequential Decision Making Problems  under Uncertainty." 2015) which adapts classical MDP models to Qualitative Possibility Theory. This theory is very close to logic  (and therefore could easily integrate the specifications framework)  while retaining properties similar to Probability Theory  (which allows planning with the same tools).  ** significance ** Although the idea and the algorithms developed are really interesting,  I doubt that the theoretical and applied contributions are worthy of NIPS.  I remain cautious however because the lack of clarity, the absence of sketch of proof  and the short time to review prevents me from checking developments in details.