Applying reinforcement learning (RL) to real-world problems will require reasoning about action-reward correlation over long time horizons. Hierarchical reinforcement learning (HRL) methods handle this by dividing the task into hierarchies, often with hand-tuned network structure or pre-defined subgoals. We propose a novel HRL framework TAIC, which learns the temporal abstraction from past experience or expert demonstrations without task-specific knowledge. We formulate the temporal abstraction problem as learning latent representations of action sequences and present a novel approach of regularizing the latent space by adding information-theoretic constraints. Specifically, we maximize the mutual information between the latent variables and the state changes. A visualization of the latent space demonstrates that our algorithm learns an effective abstraction of the long action sequences. The learned abstraction allows us to learn new tasks on higher level more efficiently. We convey a significant speedup in convergence over benchmark learning problems. These results demonstrate that learning temporal abstractions is an effective technique in increasing the convergence rate and sample efficiency of RL algorithms.
Summary:
This paper develops a method for learning a latent action representation based on prior experiences (and specifically, prior action sequences). Additionally, the paper proposes to regularize the learning of this representation using an information-theoretic constraint, yielding Temporal Abstraction with Information-theoretic Constraints (TIAC). Indeed, one promise of HRL is to allow for learning and decision making algorithms to take the long-term consequences of a decision into account when planning, exploring, assigning credit, or simply acting. The options framework (Sutton, Precup, and Singh; 1999) is a promising and well-studied toolkit for investigating these capacities of HRL. For this reason, the topic of the paper is well chosen: continuing to understand how options can benefit and accelerate RL in rich environments is an important direction for research. The idea at the core of the paper is new to my knowledge: learning an encoding of action sequences with a continuous latent representation. It could be a promising technique for HRL. Experiments are conducted to evaluate the effectiveness of the method in several environments, including a continuous gridworld, control tasks, and problems involving transfer learning. 
Verdict: Due to lack of clarity in describing the main methods, and missing comparison to any HRL/option baselines, I recommend rejection.
More Detail:
The paper is lacking clarity in its current form. I view the main contribution as the development of the architecture and loss function that together learn an appropriate latent action representation. There are two key issues with clarity at present: 1) The presentation of the core technical contributions could be improved (see comments below in "Q1"), and 2) Motivation for this style of option learning is missing, with evidence that the proposed method is in fact learning an appropriate thing.
Toward (1): I provide suggestions where clarity could be improved below in "Q1'.
Toward (2): There are a few aspects of the motivation that could be improved. First, the paper mentions that the learned options/representation will help in planning, but planning is not studied in the paper. For example: "Further, the interpolations between two sequences smoothly transfer from one to the other, which is a desired property to have during planning, because the smooth option space provides the RL algorithm with a better search space." By my reading of the paper, this claim is not studied. Similarly, in the intro: "...allow us to do planning at a higher level, and easily transfer the knowledge between different tasks". Including experiments that explicitly evaluate the capacity of the learned representation to carry out planning would help support these claims. Or, alternatively, the contribution could be focused to model-free and policy-based learning, which is where the empirical evidence currently offers the most support. Second, no HRL baselines are compared to in the experiments. One natural comparison to include would be to the Option-Critic, which was the first technique for combining option learning with deep RL. To determine whether TIAC is a sensible approach to learning and using options, a comparison to at least one other option learning method is needed. The paper currently highlights the fact that the option-critic requires a pre-specified number of options: this is true, but it is not discussed why is this problematic, or how the current proposal remedies this difficulty. Others that may be relevant include FuN (Vezhnevets et al. 2017), the recent methods of Nachum et al. (2018), among others (Tiwari and Thomas 2019, Harb et al. 2018, Harutyunyan et al. 2019, Levy et al. 2019).
In short: the results here are promising, so I encourage the authors continue in this direction. The paper will be improved if the presentation of Section 3 is sharpened (see questions regarding clarity below) and a comparison with relevant baselines is included. 
Main Questions:
Q1: The exposition of the main method (Section 3) was unclear to me. Here are a few questions I was left with:
	(a) Why is the posterior (on $o$) conditioned only on the action history, and not state?
	(b) Additionally $o$ is being treated as a random variable through 3.2. So, what is $o$? Where is the randomness coming from?
	(c) Section 3.3 states "it encodes the action sequences with respect to the L2 distance in action space". Does this mean the action space is always a subset of $\mathbb{R}$? But, it looks like $\mathcal{A}$ is just defined as some set: in Section 3.1, "$\mathcal{A}$ is the action set". So, I am confused as to what the $L_2$ is distance defined with respect to. If the actions are always assumed to be real numbers that is entirely okay, but it would be helpful to have that stated early on. From the additional text in Section 3.3, it sounds like the transition function of each action is involved in computing this distance ("...only have small difference in each step of action. Due to the error compounding, the two sequences..."). 
	(d) How is the estimate of the posterior actually used to act? The output of "D" in Figure 2 is $\hat{a_{0...k}}$. What is the type of this entity? Is it guaranteed to be an element of $\mathcal{A}$? If so, then the "option" here is a policy that maps $o$ and the action history $a_{0...k}$ to a new action, correct? Ah, so in Figure 3, it looks like D will have different output depending on how the termination condition is handled. Are the actions output by $D$ then executed by the RL agent, or is there some additional decision making that goes on downstream?
	(e) Early on the section states "In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing  the information which could benefit RL training." It might be helpful to include some intuition about what this information would look like. It's unclear why action history would be all that meaningful on its own (without say, the state history). It would help the section to provide some intuition for such a latent variable existing; is there an idealized, simple case that would help convey the idea? Note that this proposal comes across as different from the original proposal of the options framework: As an example, Sutton, Precup, and Singh (1999) say: "options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way". This temporally abstract knowledge need not be a function of the entire action history. I like this aspect of the method as it makes the proposed algorithm quite novel, but the motivation for why this should work didn't come through for me.
	(h) Should the mutual information in Eq. 4 be the conditional mutual information given $a_{0...k}$? (Same question for the remaining uses of $I$ and $H$).
	(i) It is unclear how the option learning coordinates with the RL algorithm used. That is, suppose we train the HRL component to learn the mapping from $s, a_{0...k}$ to the constituents identified in Figure 2/3. Where does the actual RL take place? Does the algorithm just execute the actions output by $D$ at each time step? 
Q2: In the first experiment, it is stated: "because the smooth option space provides the RL algorithm with a better search space." Any thoughts as to why this is true? Including some discussion here might help motivate the approach.
Minor Comments:
	C1: I do not understand Figure 6. The color is said to denote "the distribution of options", but I couldn't quite make out what this was, precisely. It would be helpful to know the range of values it can take on, and how those values map to the displayed colors. Moreover, what is the take away from the figure? The text states "with information-theoretic constraints the options and state changes become more correlated" but I am having trouble connecting that claim with the visuals themselves. Some additional discussion here would be really helpful.
	
	C2: In Figure 5, what does "dimension disturbance in option space" mean?
Minor Typos/Writing Suggestions [did not affect evaluation]:
	Abstract:
	- "Applying reinforcement learning (RL) to"::"Applying reinforcement learning (RL) algorithms to"
	- I am having trouble parsing this phrase: "to learn new tasks on higher level more efficiently". Perhaps: "to learn new tasks at a higher level of abstraction more efficiently"
	- "over benchmark learning problems"::"over baseline learning algorithms on benchmark problems"
	Sec. 1 (Intro):
	- Plural acronyms tend to have an 's' at the end. So: Recurrent Variational AutoEncoders (RVAEs).
	- "conveys meaningful information and benefit the RL training"::"conveys meaningful information and can benefit learning"
	Sec. 2 (Related Work):
	- "the policy sketches"::"policy sketches"
	- Personal preference, by I always prefer "use" to "utilize".
	Sec. 3 (Approach):
	- Your $\mapsto$ operators should be replaced by $\rightarrow$. The $\mapsto$ operator indicates what is applied to elements on the left, while $\rightarrow$ specifies the domain and codomain of the function. Thus, the $\mapsto$ variation would be $P : (s,a) \mapsto s'$. The story is the same for $\beta$: it should read "$\beta : \mathcal{S} \rightarrow [0,1]$". Note that this (using $\rightarrow$) is how Sutton, Precup, and Singh (1999) define $\beta$ as well.
	- "Sub-policy is defined as a function over the random variable."::"Now, the sub-policy is defined as a function over the random variable."
	- Not a sentence: "So that the options with similar consequences become closer in the option space." Consider combining with the previous sentence.
	- This sentence runs on: "Given a set of past experiences...". Consider defining $\Lambda$ first as its own sentence, then definines the problem. Something like: "We let $\Lambda = ...$ Then, our problem is to learn...".
	- "it is empirically shown"::"it has been demonstrated empirically"
	- Latex quote issue: "”go reach the door".
	- In Equations 4-9: in general, mutual information is a function of random variables. Is $o$ a random variable? For instance I have trouble expanding $H(o)$. What is $p(o)$?
	- "the encode $E$ is regularized"::"the encoder $E$ is regularized"
	Sec. 4 (Experiments):
	- "task for proof of the concept"::"task as a proof of concept"
	- "that allows us easily visualize the option we learned from the experience"::"that allows us to easily visualize the options learned from experience"
	- "that the RVAE nicely capturing the direction":::"that the RVAE captures the direction"
	- Misuse of $\mapsto$: "learn a control policy $\pi : \mathcal{S} \mapsto \mathcal{A}$" should be "learn a control policy $\pi : \mathcal{S} \rightarrow \mathcal{A}$" or "learn a control policy $\pi : s \mapsto a$".
	- "HarfCheetah"::"HalfCheetah"
References:
Vezhnevets, Alexander Sasha, et al. "Feudal networks for hierarchical reinforcement learning." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in Neural Information Processing Systems. 2018.
Tiwari, Saket, and Philip S. Thomas. "Natural option critic." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
Harb, Jean, et al. "When waiting is not an option: Learning options with a deliberation cost." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Harutyunyan, Anna, et al. "The Termination Critic." AISTATS 2019
Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." ICLR 2019.