Mutual Information between agent Actions and environment States (MIAS) quantifies the influence of agent on its environment. Recently, it was found that the maximization of MIAS can be used as an intrinsic motivation for artificial agents. In literature, the term empowerment is used to represent the maximum of MIAS at a certain state. While empowerment has been shown to solve a broad range of reinforcement learning problems, its calculation in arbitrary dynamics is a challenging problem because it relies on the estimation of mutual information. Existing approaches, which rely on sampling, are limited to low dimensional spaces, because high-confidence distribution-free lower bounds for mutual information require exponential number of samples. In this work, we develop a novel approach for the estimation of empowerment in unknown dynamics from visual observation only, without the need to sample for MIAS. The core idea is to represent the relation between action sequences and future states using a stochastic dynamic model in latent space with a specific form. This allows us to efficiently compute empowerment with the "Water-Filling" algorithm from information theory. We construct this embedding with deep neural networks trained on a sophisticated objective function. Our experimental results show that the designed embedding preserves information-theoretic properties of the original dynamics.
This paper proposes to take advantage of a known result on the channel capacity of a linear-Gaussian channel in order to estimate the empowerment and maximize mutual information between  policies (action sequences) and final states (given  initial states). The idea is to map the raw action sequences and states to a latent space where learning would force that linear property to be appropriate.
I like the general idea of the paper (as stated above) along with its objectives but I have several concerns.
First, I need to be reassured that we are computing the right quantity. Channel capacity is the maximum mutual information (between inputs and outputs) over the input distribution, whereas I had the impression that empowerment would be this mutual information, and that we want to increase it, but not necessarily reach its maximum over all possible policies: it would usually be one of the terms in an objective function (e.g. here we have reconstruction error, and in practice there would be some task to solve in addition to the exploration reward). One way to see this problem in the given formulation is that the C* objective only depends on the matrix A (which encapsulates the conditional density of z_{t+1} given the trajectory) and it does not depend at all on the distribution of the trajectory itself! This is weird since if we are going to use this as reward the objective is to improve the trajectory. What's the catch? So either I misunderstand something (which is quite possible) or there is something seriously wrong here.
I am assuming that the training objective for the encoder is a sum of the reconstruction error and of C*. But note how this does not give a reward for policies, as such. This is a bit strange if the goal is to construct an exploratory reward!
A less radical comment is: have you verified that the linear relationship between z_{t+k} and the b actually holds well? In other words, is the encoder able to map the raw state and actions to a space where the linearity assumption  is correct, and thus where equation (3) is satisfied.
Figure 3 has something weird,  probably one of the two sequences of -1's should be a sequence of +1's.
Figure 4 is difficult to interpret, the caption should do a better job.
The experiment on the safety of the RL agent is weak. I don't see the longer path as safer, here. And the results are not very impressive, since the agent is only doing what it's told, i.e., go in areas with more options (more open areas) but there is no reason to believe that this is advantageous, here.
Finally, what would make this paper much more appealing is if the whole setup led to learning better high-level representations (how to measure that is another question, but it is a standard kind of question in representation learning papers).
Related work:
I don't understand why in the abstract the authors refer to sampling-based methods as requiring exponentially many samples. This is not generally the case for sampling based methods (e.g. think of VAEs). I suppose the authors refer to something in particular but it was not clear to me what.
References: in the intro, you might want to refer to the contrastive methods and variational methods to maximize mutual information between representations of the state and representations of the actions, e.g.,
 Thomas et al, 2018, 1802.09484
 Kim et al, 2018, arXiv:1810.01176
 Warde-Farley et al, 2018, arXiv:1811.11359
 
Evaluation: for now I suggest a weak reject but I am ready to modify my score if I am convinced that my main concerns were unfounded.