Mixed-precision arithmetic combining both single-and half-precision operands on the same operation has been successfully applied to train deep neural networks. Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register file size, it has a limited capacity for diminishing computing costs and requires 32-bits to represent its output operands. This paper proposes two approaches to replace mixed-precision for half-precision arithmetic during a large portion of the training. The first approach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training. The second approach reaches the same accuracy as the state-of-the-art by dynamically switching between half-and mixed-precision arithmetic during training. It uses half-precision during more than 94% of the training process. This paper is the first in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy.
This proposes two techniques to replace mixed-precision arithmetic with half-precision training for a large part of the training process. In the first approach, the authors simply switch all mixed-precision operations with half-precision operations, and can achieve performances slightly lower than SOTA. In the second approach, the authors propose to dynamically switch between mixed-operations and half-precision operations during training. The authors claim that this second approach can match SOTA results while using half-precision arithmetic for more than 94% of training.
Overall the paper is fairly well written, and easy to follow. The proposed techniques seem to empirically work well. However, I have a number of concerns about the paper, which explains my score. I list these concerns below.
1. The proposed approach has a number of additional hyperparameters, which makes it less likely for the algorithm to be widely used if the algorithm is very sensitive to the values of this. For very extreme values of these hyperparameters, I would expect the algorithm to start behaving quite poorly. But it would help a lot to provide some sensitivity analysis to these hyperparameters for reasonable values of these hyperparameters.
2. How much do the optimal hyperparameters (like numBatchesMP, numBatchesBF16, emaT) vary across problems?
3. How much do the above-mentioned optimal hyperparameters vary with mini batch size?
4. How are the other hyperparameters like learning rates selected? Is the learning rate tuned?
5. Are the experiments repeated multiple times?
6. It seems a bit weird to call a modification that simply uses half-precision arithmetic for most FMA operations a significant contribution of the paper, especially since it can't reach SOTA performance.
7. Algorithm 1 should be written out in a better way that shows the training loop. It is slightly confusing the way it is written up right now.
Overall I think the paper would significantly benefit from a more thorough empirical evaluation.
=============================
Edit after rebuttal:
I thank the reviewers for their response and for running the additional experiments. However, I find the updated version of the paper to be inadequate in fully answering my concerns. While the authors have included a hyperparameter sensititivity analysis, I find the experiment to be unconvincing. Only two of the three hyperparameters are swept over a very small range of values, and the results presented are only for the first 12 epochs, while the actual model is typically trained for 90 epochs. While I appreciate the added experiment and realize that 10 days is too short a time to put in a proper sensitivity analysis, based on the current draft of the paper, I cannot recommend accepting this paper. I am however raising my score to a weak reject.