The information bottleneck principle in (Tishby et al., 2000)  is an elegant and useful approach to representation learning. In this paper, we investigate the problem of representation learning in the context of reinforcement learning using the information bottleneck framework, aiming at improving the sample efficiency of the learning algorithms. We analytically derive the optimal conditional distribution of the representation, and provide a variational lower bound. Then, we maximize this lower bound with the Stein variational (SV) gradient method (originally developed in (Liu & Wang, 2016; Liu et al., 2017)). We incorporate this framework in the advantageous actor critic algorithm (A2C) (Mnih et al., 2016)  and the proximal policy optimization algorithm (PPO) (Schulman et al., 2017). Our experimental results show that our framework can improve the sample efficiency of vanilla A2C and PPO significantly. Finally, we study the information bottleneck (IB) perspective in deep RL with the algorithm called mutual information neural estimation(MINE) (Belghazi et al., 2018). We experimentally verify that the information extraction-compression process also exists in deep RL and our framework is capable of accelerating this process. We also analyze the relationship between MINE and our method, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound.
This paper proposes a representation learning algorithm for RL based on the Information Bottleneck (IB) principle. This formulation leads to the observed state X being mapped to a latent variable Z ~ P(Z | X), in such a way that the standard loss function in actor-critic RL methods is augmented with a term minimizing the mutual information between X and Z (which can be seen as a form of regularization). This results in a loss that is difficult to optimize directly in the general case: the authors thus propose to approximate it through a variational bound, using Stein variational gradient descent (SVGD) for optimization, which is based on sampling multiple Z_i’s for a given state X, so as to compute an approximate gradient for the parameters of the function mapping X to Z. Experiments show that when augmenting the A2C algorithm with this technique, (1) the mutual information I(X, Z) decreases more quickly (better « compression » of the information), and (2) better sample efficiency is observed on 5 Atari games (with also encouraging results with PPO on 3 Atari games).
In spite of the interesting theoretical contributions, I have to recommend rejection as the current empirical evaluation of the proposed approach is extremely limited, making it difficult to assess its benefits over more straightforward algorithms.
On the positive side, the authors derive a sensible approach to IB representation learning in RL, and provide solutions to the optimization challenges it leads to. I did not have time to check all the maths in the Appendix (I only went through the derivations in A.1 and A.2), but they seem to make sense overall (though it is unclear to me if the new algorithm proposed in A.5 is a practical one, so I am not taking it into account in this evaluation).
The key negative point is definitely the weak empirical evaluation. The main results are from a limited sample of 5 Atari games, when the full Atari benchmark has 10x more games and is known to exhibit high variance among games when comparing RL algorithms (the additional results from the Appendix on 3 additional games also show situations where the proposed method does not seem to help much, confirming that larger scale experiments are needed for a proper evaluation). In addition it seems like each algorithm is run only once (instead of using multiple seeds) and only over ~200K timesteps, which is three orders of magnitude lower than results typically reported on Atari. Another issue is that there is no comparison to other representation learning techniques (like those mentioned in the related work section, or the recent "Unsupervised State Representation Learning in Atari"), nor to a natural and more straightforward variant of the proposed method where Z would simply be sampled from a (learned) Gaussian distribution Z ~ N(mu(X), var(X)), which at first sight seems like an easier-to-optimize objective (using the reparameterization trick)… I may be wrong, but then this should probably be explained in the paper (I realize that the proposed approach is more general, but then it should be shown how this extra flexibility can lead to improved results). Finally, the impact on runtime performance is not analyzed: how much slower do A2C / PPO become when optimizing the mutual information term with SVGD? Overall it is really unclear that better RL results can be obtained through this technique.
Another important issue is that I found the paper rather difficult to follow, due to some inconsistent / unclear notations or equations. Here are the main ones I noted:
•	The discount factor is not accounted for in the derivation of the objectives in eq. 1-2 (I know this is often the case in practice, but the reason for dropping it should at least be mentioned)
•	The jump from V^pi(X) to V^pi(Z) at the end of Section 3 is explained too succintly. It suggests that V^pi(Z) must be constant (equal to V^pi(X)) over all values of Z that may be sampled from X, which as far as I can tell is not the case in the rest of the paper. It is also unclear whether pi depends on Z or X. And the notation J(Z) makes it look like J does not depend on X, but it seems like it does because even if pi depends only on Z, by its definition V^pi(Z) actually still depends on X (this also leads to weird equations like eq. 33 where X does not appear in the right-hand side). Overall this is rather confusing.
•	The « for every X » at the top of p. 4 does not make sense to me, due to the term I(X, Z) in eq. 3 where X is a random variable and thus does not take a specific value.
•	The paragraph below eq. 4 is a bit confusing. It looks like it amounts to saying that Y is a Gaussian around V(Z)?
•	Eq. 5 suggests that R depends on Z, but shouldn’t it depend on X? If it depended on Z, then wouldn’t it influence the optimization since by modifying P(Z|X) we can control the distribution on Z and thus the distribution on R?
•	In eq. 10 it is unclear whether L1 and L2 contain the expectation, also L2 is defined as a function of both theta and phi but seems to depend only on theta
•	Below eq. 15 it is said that « P is the distribution of Z » but P does not appear in eq. 15
•	In eq. 16 should the phi on the left hand side be phi* as in eq. 15?
•	Below eq. 16 it is said « Notice that C has been omitted », but it is unclear whether it was not included to alleviate notations or because it disappears naturally in the mathematical derivation of eq. 16
•	The motivation for introducing zeta below eq. 17 is unclear, especially since it seems to play an important role considering that zeta = 0.005 << 1 is used in the experiments (with no explanation as to how this specific value was selected)
•	\hat{L}(Z_i, theta, phi) (between eq. 17 and 18) does not seem to be defined
•	What is the motivation for using the Gaussian U(Z) as described in Section 5? In particular I find it weird that it depends on X_i, while U(Z) is supposed to replace the marginal P(Z) and not the conditional P(Z | X)
Minor points:
•	In the definition of Y_t = R_t = sum_i=0^n-2 … above eq. 4, I think the sum should be up to n-1 for an n-step return
•	Eq. 4 uses R_t on the right hand term instead of Y_t which looks weird
•	Theorem 1 states « Assume that for any epsilon > 0, … »: « for any » should probably be « there exists », since if the inequality was true for any epsilon, it would imply both mutual informations are equal (also the formulation of the theorem does not make it clear that the last inequality is the main result)
•	Footnote 1 p. 4: I(X, Y) should be I(X, Z)
•	In eq. 15 the phi below the argmax should be in bold
•	Please add a reference that the reader can refer to in order to understand where eq. 20 is coming from
•	« Apparently » is used in a couple of places but should probably be replaced with another word
•	When U is uniform, how do you choose its support?
•	After « median of pairwise distances between the particles », I think i=1 should be j=1
•	It is unclear to me what « A2C with noise is »: it is said that the same phi(X, epsilon) is used « as A2C with our framework », but is it the same phi as A2C with uniform SVIB or A2C with Gaussian SVIB? And whichever it is, how does it differ from the one it is equal to?
•	« we add 21 to all four curves in order to make exponential moving average »: I do not understand that sentence
•	« we set the number of samples as 26 for the sake of computation efficiency »: I fail to see how going from 32 to 26 is going to make a major difference in computational efficiency
Update: thank you for your response, but in the absence of a revised version addressing my concerns (as well as those from the other reviewers), I cannot increase my rating