An obstacle that prevents the wide adoption of (deep) reinforcement learning (RL) in control systems is its need for a large number of interactions with the environment in order to master a skill. The learned skill usually generalizes poorly across domains and re-training is often necessary when presented with a new task. We present a framework that combines techniques in formal methods with hierarchical reinforcement learning (HRL). The set of techniques we provide allows for the convenient specification of tasks with logical expressions, learns hierarchical policies (meta-controller and low-level controllers) with well-defined intrinsic rewards using any RL methods and is able to construct new skills from existing ones without additional learning. We evaluate the proposed methods in a simple grid world simulation as well as simulation on a Baxter robot.
The paper argues for structured task representations (in TLTL) and shows how these representations can be used to reuse learned subtasks to decrease learning time.
Overall, the paper is sloppily put together, so it's a little difficult to assess the completeness of the ideas. The problem being solved is not literally the problem of decreasing the amount of data needed to learn tasks, but a reformulation of the problem that makes it unnecessary to relearn subtasks. That's a good idea, but problem reformulation is always hard to justify without returning to a higher level of abstraction to justify that there's a deeper problem that remains unchanged. The paper doesn't do a great job of making that connection.
The idea of using task decomposition to create intrinsic rewards seems really interesting, but does not appear to be explored in any depth. Are there theorems to be had? Is there a connection to subtasks rewards in earlier HRL papers?
The lack of completeness (definitions of tasks and robustness) also makes the paper less impactful than it could be.
Detailed comments:
"learn hierarchical policies" -> "learns hierarchical policies"?
"n games Mnih et al. (2015)Silver et al. (2016),": The citations are a mess. Please proof read.
"and is hardly reusable" -> "and are hardly reusable".
"Skill composition is the idea of constructing new skills with existing skills (" -> "Skill composition is the idea of constructing 
new skills out of existing skills (".
"to synthesis" -> "to synthesize".
"set of skills are" -> "set of skills is".
"automatons" -> "automata".
"with low-level controllers can" -> "with low-level controllers that can".
"the options policy π o is followed until β(s) > threshold": I don't think that's how options were originally defined... beta is generally defined as a termination probability.
"The translation from TLTL formula FSA to" -> "The translation from TLTL formula to FSA"?
"four automaton states Qφ = {q0, qf , trap}": Is it three or four?
"learn a policy that satisfy" -> "learn a policy that satisfies".
"HRL, We introduce the FSA augmented MDP" -> "HRL, we introduce the FSA augmented MDP.".
" multiple options policy separately" -> " multiple options policies separately"?
"Given flat policies πφ1 and πφ2 that satisfies " -> "Given flat policies πφ1 and πφ2 that satisfy ".
"s illustrated in Figure 3 ." -> "s illustrated in Figure 2 ."?
", we cam simply" -> ", we can simply".
"Figure 4 <newline> ." -> "Figure 4.".
", disagreement emerge" -> ", disagreements emerge"?
The paper needs to include SOME definition of robustness, even if it just informal. As it stands, it's not even clear if larger 
values are better or worse. (It would seem that *more* robustness is better than less, but the text says that lower values are 
chosen.)
"with 2 hidden layers each of 64 relu": Missing word? Or maybe a comma?
"to aligns with" -> "to align with".
" a set of quadratic distance function" -> " a set of quadratic distance functions".
"satisfies task the specification)" -> "satisfies the task specification)".
Figure 4: Tasks 6 and 7 should be defined in the text someplace.
"current frame work i" -> "current framework i".
" and choose to follow" -> " and chooses to follow".
" this makes" -> " making".
"each subpolicies" -> "each subpolicy".