Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training. Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages. We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter. We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods. We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy compared to the existing state-of-the-art MPT and single-precision floating point.
The authors propose an adaptive loss scaling method during the backpropagation stage for the mix precision training to reduce the underflow. Compared with the previous work, which scales the loss by human design, and needs to be consistent in all layers. The authors state that they can decide the scale rate layer by layer automatically to reduce the underflow in a low precision situation. 
They calculate the scale rate using the statistic information of the layer weight and gradient. By adaptively scale each layer’s loss and gradient, this method can reduce the underflow rate better than the previous work. Additionally, the authors claim that the computation overhead is not significant, so it is efficient to use rather than searching from a set of fix scale rates. 
The experiments present on image classification and objective detection benchmarks. From the result, we can see that the adaptive loss scale reaches a relatively high point on all the tasks.
 
Pros:
 
- The method is straight forward and easy to understand. The motivation is good. They get some impressive results on ResNet110 and SSD512 comparing with the fixed scaling method. Besides, they give some analysis of their advantages and disadvantages in different networks, which looks promising to me.
 
Cons:
- One question in Section 3.2.1, the assumption that w, g, p can be treated as the random variable with Gaussian distribution seems not natural in the training process. Especially p is a zero-mean distribution. The cited paper uses this assumption in a more convincing case, such as the weight initialization task. Notice that He et al., 2015 claim that the product of weight and gradient can be a zero-mean normal distribution is based on the weight is a symmetric distribution around zero, which is not true in neither this paper’s assumption nor the real training situation. 
- In the objective detection part, I can not find which dataset the authors use. Though the author state that they follow Liu et al., 2016 ’s work, there are also several tasks in Liu’s paper, and I can not directly match the resulting point with any of those tasks, which makes me hard to confirm the experiment result.
- The experiment setting is unclear. Here are two questions. 1, What is the initial scale at the last layer? It should be manually designed in the experiment, and I think this value may affect the other layer’s scale as well. If the algorithm is robust for this scale, it is better to show some study on that. 2, What update frequency is used in the experiment? The authors say that the overhead can be reduced by reducing the frequency, but they do not clearly show which frequency they use in their experiment, if the frequency does not affect the performance, it is also better to claim or show some study on that.
Minor comments:
- Figures can use a larger font.  Figures 4a and 4b can be aligned better.