Batch size selection affects the generalization ability of gradient descent, and small batch size is usually preferred. Conventionally, weight regularization is performed for every mini-batch without considering when would be the right time to regularize weights during optimization steps. For model compression, which regularizes weights to follow compressed forms, compression-aware training also performs weight compression for every mini-batch to compute the impact of compression on the loss function. In this paper, we propose a new hyper-parameter called "Non-Regularization period" or NR period during which weights are not regularized (or compressed). We first investigate the influence of NR period on regularization using weight decay and weight random noise insertion. Throughout various experiments, we show that stronger weight regularization demands longer NR period (regardless of batch size) to best utilize regularization effects. From our empirical evidence, we argue that weight regularization for every mini-batch allows small weight decay/noise only and limited regularization effects such that there is a need to search for right NR period and weight regularization strength to enhance model accuracy. Consequently, NR period becomes especially crucial for model compression where strong weight regularization is necessary to increase compression ratio. Using various models, we show that simple weight regularization to comply with compression formats along with long NR period is enough to achieve high compression ratio and model accuracy.
This work investigates compression-aware training and introduces a new hyper-parameter called "Non-regularization period". Basically, the paper proposes to apply weight regularization and compression less frequently so that they can use stronger regularization/compression, hence achieving high compression ratio and model accuracy.
Overall, the idea of asynchronous regularization and compression is interesting and worth further investigation. However, I vote for a reject for now because
(1) The paper is hard to follow and the writing can be significantly improved especially for the abstract and introduction. I felt rather confused when I first read the abstract and introduction. The authors frequently use the word "weight update", however it is unclear whether it refers to gradient update or the update caused by regularization/compression. Besides, the title is also a little confusing. There's little discussion about the choice of batch size and I don't really understand how weight regularization is decoupled from batch size after reading the paper. I suggest the authors to think more about the title.
(2) The reason why such a training scheme improves model compression is unclear and needs further investigations. In the paper, the authors first interpret model compression as a way of inducing weight decay (and random weight noise). Particularly, the model accuracy is roughly constant over different value of NR period (as shown in Figure 3 and Figure 4) when weight decay is used. However, the results of model compression are quite different (according to Figure 5). The best performance is only achieved with very large NR period in the case of SVD and pruning. I think the authors should give some explanations for that difference. Also, I notice that quantization behaves more similarly to weight decay and it requires further discussion.
To sum up, I like the idea of asynchronous regularization/compression, but I'm not quite satisfied with current version of paper. I encourage the authors to improve their writing and add more discussions (see my point (2)) to the paper. I'm willing to increase my score if the authors can address my concerns.