VARIATIONAL DISCRIMINATOR BOTTLENECK: IMPROVING IMITATION LEARNING, INVERSE RL, AND GANS BY CONSTRAINING INFORMATION FLOW
Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from raw video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods. (Video  1 )
Summary:
The authors propose to apply the Deep Variational Information Bottleneck (VIB) method of [1] on discriminator networks in various adversarial-learning-based scenarios. They propose a way to adaptively update the value for the bêta hyper-parameter to respect the constraint on I(X,Z). Their technique is shown to stabilize/allow training when P_g and P_data do not overlap, similarly to WGAN and gradient-penalty based approaches, by essentially pushing their representation distributions (p_z) to overlap with the mutual information bottleneck. It can also be considered as an adaptive version of instance noise, which serves the same goal. The method is evaluated on different adversarial learning setup (imitation learning, inverse reinforcement learning and GANs), where it compares positively to most related methods. Best results for ‘classical’ adversarial learning for image generation are however obtained when combining the proposed VIB with gradient penalty (which outperforms by itself the VGAN in this case).
Pros :
- This paper brings a good amount of evidence of the benefits to use the VIB formulation to adversarial learning by first showing the effect of such approach on a toy example, and then applying it to more complex scenarios, where it also boosts performance. The numerous experiments and analyses have great value and are a necessity as this paper mostly applies the VIB to new learning challenges. 
- The proposition of a principled way of adaptively varying the value of Bêta to actually respect more closely the constraint I(X,Z) < I_c, which to my knowledge [1] does not perform, is definitely appealing and seems to work better than fixed Bêtas and does also bring the KL divergence to the desired I_c.
- The technique is fairly simple to implement and can be combined with other stabilization techniques such as gradient penalties on the discriminator.
Cons:
- In my view, the novelty of the approach is somewhat limited, as it seems like a straightforward application of the VIB from [1] for discriminators in adversarial learning, with the difference of using an adaptive Bêta.
- I think the Bêta-VAE [2] paper is definitely related to this paper and to the paper on which it is based [1] and should thus be cited as the authors use a similar regularization technique, albeit from a different perspective, that restricts I(X,Z) in an auto-encoding task.
- I think the content of batches used to regularize E(z|x) w.r.t. to the KL divergence should be clarified, as the description of p^tilde “being a mixture of the target distribution and the generator” (Section 4) can let the implementation details be ambiguous. I think batches containing samples from both distributions can cause problems as the expectation of the KL divergence on a batch can be low even if the samples from both distributions are projected into different parts of the manifold. This makes me think batches are separated? Either way, this should be more clearly stated in the text.
- The last results for  the ‘traditional’ GAN+VIB show that in this case, gradient penalty (GP) alone outperforms the proposed VGAN, and that both can be combined for best results. I thus wonder if the results in all other experiments could show similar trends if GP had been tested in these cases as well. In the imitation learning task, authors compare with instance noise, but not with GP, which for me are both related to VIB in what they try to accomplish. Was GP tested in Imitation Learning/Inverse RL ? Was it better? Could it still be combined with VIB for better results? 
- In the saliency map of Figure 5, I’m unclear as to what the colors represent (especially on the GAIL side). I doubt that this is simply due to the colormap used, but this colormap should be presented.
Overall, I think this is an interesting and relevant paper that I am very likely to suggest to peers working on adversarial learning, and should therefore be presented. I think the limited novelty is counterbalanced by the quality of empirical analysis. Some clarity issues and missing citations should be easy to correct. I appreciate the comparison and combination with a competitive method (Gradient Penalty) in Section 5.3, but I wish similar results were present in the other experiments, in order to inform readers if, in these cases as well, combining VIB with GP leads to the best performance.
[1] Deep Variational Information Bottleneck, (Alemi et al. 2017)
[2] beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework (Higgins et al. 2017)