In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.
Summary: The paper proposes training dynamics-aware embeddings of the state and k-action sequences to aid the sample efficiency of reinforcement learning algorithms. The authors propose learning a low-dimensional representation of the state space, $z_s$ as a well as a temporally extended action embedding, $z_a$. The latter will be used in conjunction with a higher level policy that plans in this abstract action-space, $z_a$. By using these two embeddings, the authors test the proposed system on a set of Mujoco tasks and show improved results.
Positives:
1) Fairly simple objective, in line with previous work on unsupervised learning methods to representation learning in RL (like DARLA, variational intrinsic control, etc).
2) The temporally extended nature of the action embedding makes is particularly attractive for HRL systems as a continuous space of options (via $z_a$).
Questions and Points of improvements:
1) Main concern: The need to pre-train the embedding before the RL task, I strongly believe limits the applicability of the proposed algorithm. The embeddings are trained under a uniformly random policy, which in many cases in RL is not informative enough to reach, with decent probability, many of the states of interest. Thus the embedding will reflect only a small subset of the state/action-space. Thus it will be highly depend on the tasks under consideration if this is enough variety for generalisation across the stationary distribution of more informed RL policies. Implicitly, the authors are making a continuity assumptions over the state and action space.
(To be more precise: A particular failure case of the action embedding would be if say one of the action (down) has no effect in the part of the space where the uniform policy has explored. Now this becomes an important action in a level down the line where the agents needs to go down a tunnel -- example from Atari's Pitfall. In this case, under the embedding training, since the down has had no effect in training, this action will not be represented at all. This would means the RL algorithm could not ever learn to use it). 
The co-evolution of the representation and the RL policy, I think it's paramount especially when dealing with exploration.
2) Q: Section 3.2: "we extend... to work with temporally extended actions while maintaining off-policy updates ..". Can the authors expand on how this is done? Both updates in this section seem to be on policy ($\mu$).
3) Q: Section 3.2: "Q can only be trained on $N/k$ observations. This has a substantial impact on sample efficiency". Note that this is actually an explicit trade-off between reduced number of samples we see ($N/k$) and the increased horizon in propagating information, due to the effective k step update. This trade-off need not be optimal for $k=1$.
4) Notes of experiments:
a) It is hard to assess the difficulty of the exploration problems investigated. This relates to point 1) and the implicit assumptions highlighted there. 
b) It would have been nice to have a study on $k$ and it's impact on the sample complexity. The larger the $k$ the harder the representation learning problem becomes; and possibly the larger the number of samples needed to learn in this combinatoric space. How does this trade-off with the benefits one could potentially get in the RL phase?
c) For the comparison algorithms: where any of these using a temporal extended update rule?  Or are all of them 1-step TD like algorithms? It would good to separate the effect of the multiple-step update in Sec. 3.3 and the exploration in this abstract action space.