CAT: Compression-Aware Training for bandwidth reduction
Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference. Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization. For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper.
In this article, the authors propose a compression-aware method to achieve efficient compression of feature maps during the inference procedure. When in the training step, the authors introduce a regularization term to the target loss that optimizes the entropy of the activation values. At inference time, an entropy encoding method is applied to compress the activations before writing them into memory. The experimental results indicate that the proposed method achieves better compression than other methods while holding accuracy.
There are still some issues as follows:
1.	The authors should carefully check the format of the references in the whole article. For example, in section 2, line 5 from the top and line 8 from the bottom, “Xiao et al. (2017), Xing et al. (2019)” and “(Chmiel et al., 2019)” are in the wrong format.
2.	It is suggested that the authors swap the order of formulation (8) and (9) in section 3.2 so that it will be a good correlation with the formulation (3) and (4).
3.	I am interested in learning the time taken by the proposed method during the inference procedure vs other related methods.
4.	The authors studied two differentiable entropy approximation in the paper, and they stated that they calculate soft entropy only on the part of the batch for the reduction of both memory requirements and time complexity in training. I hope the authors will clarify 1) Whether the accuracy will be affected by other differentiable entropy approximations; 2) what is the impact on accuracy if only part of the batch is considered.