In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments. The options framework provides formalism for such abstraction over sequences of decisions. However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable. Indeed, it is preferable to learn options directly from interaction with the environment. Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes. In this work we develop a novel policy gradient method for the automatic learning of policies with options. This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels. Experimental results show that the options learned are interpretable. Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options.
The paper presents a new policy gradient technique for learning options. The option index is treated as latent variable and, in order to compute the policy gradient, the option distribution for the current sample is computed by using a forward pass. Hence, a single sample can be used to update all options and not just the option that has been used for this sample.
 
The idea of the paper is good but the novelty is limited. As noted by the authors, the idea of using inference for option discovery has already been presented in Daniel2016. Note that the option discovery process is Daniel2016 is not limited to linear sub-policies, only the policy update strategy is. So the main contribution is to use a new policy update strategy, i.e., policy gradients, for inference based option discovery. Thats fine but should be stated more clearly in the paper. The paper is also written very well and the topic is relevant for the ICLR conference. 
However, the paper has two main problems:
- The results are not convincing. In most domains, the performance is similar to the A3C algorithm (which does not use inference based option discovery), so the impact of this paper seems limited. 
- One of the main assumptions of the algorithm is wrong. The assumption is that rewards from the past are not correlated with actions in the future conditioned on the state s_t (otherwise we would always have a correlation) ,which  is needed to use the policy gradient theorem. The assumption is only true for MDPs. However, using the option index as latent variable yields a PoMDP. There, this assumption does not hold any more. Example: Reward at time step t-1 depends on the action, which again depends on the option o_t-1. Action at time step t depends on o_t. Hence, there is a strong correlation between reward r_t-1 and action a_t+1 as o_t and o_t+1 are strongly correlated. o_t is not a conditional variable of the policy as it is not part of the state, thats why this assumption does not work any more.
Summary: The paper is well written and presents a good extension of inference based option discovery. However, the results are not convincing and there is a crucial issue in the assumptions of the algorithm.