Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute-and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a selfsupervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
In this paper, the authors tackle the task of compressing a network. While there
are many effective solutions so far regular computer vision tasks, as they demonstrate,
they fail catastrophically  when applied to generative adversarial networks(GANs).
They propose a modification to the classic distillation method, where a
"student" network tries to imitate the uncompressed one under the supervision of
a fully converged discriminator network. They perform evaluation on multiple
tasks from image synthesis to super-resolution. They also study the influence
of the compression factor on the quality of the generated images.
The task is well motivated and situated in the related literature. The first
section is very thorough and extremely efficient at describing the failure modes
of existing methods. On one side, the results demonstrated in the evaluation are
compelling, on the other side, the compression factor is only 50%, which is much
lower than seen in related work. However, as it is shown in section 3 the task
may be much harder for GANS than regular models so I still consider it a 
sizeable contribution.
There are a couple of points that require clarification. I personally found the
description of the method (Section 4) rather confusing. It is clear
what "discriminative loss" is as it is the one used in every GAN.
Unfortunately, I could not understand what "generative loss" means in the general
case. An example is given for StarGAN in equation (7) and I have a rough idea of
what to choose for Style Transfer, Domain Translation, Super Resolution and
Image translation. Though, it is unclear to me what to use in the case of
image synthesis. The experiments clearly show that it is possible so I think
it is necessary to show how this framework is concretely applied to each task at
hand.
During training, the discriminator only ever saw pictures from the true distribution
and the distribution generated by the generator (at each of its training steps).
If I understood the framework properly, here, the compressed generator is trained
from a random initialization. The distribution it outputs is therefore
completely unknown and potentially non overlapping with either of the true or
the generator ones. In that case it is hard to predict what the discriminator
would do on completely out of distribution samples. I seems reasonable to
conjecture that it might consider them "true" because it was never trained on
them. Could you provide an explanation of why it is not a problem in practice?
Do you have to try multiple initializations? Is the generative
loss enough to force the compressed discriminator to match the support
of the distribution of the dense generator?
I think this paper is novel, tackles a hard task and presents compelling results
(albeit using very mild compression ratios). It should be accepted if some
clarifications are made in section 3.
Minor Remarks:
- Figures 2, 9, 42 and 43 are unreadable when printed with a regular color office
  printer.
- It is unclear what it would take an extra 10% of the original number of epochs to train the compressed network. Why couldn't it be faster, or much longer?