In Defense of Pure 16-Bit Floating-Point Neural Networks
The precision used should match the requirements of the dataset, the training process, and the available compute. There are practical uses to 16-Bit FP training.<p>&quot;Our findings demonstrate that pure 16-bit floating-point neural networks can achieve similar or even better performance than their mixed-precision and 32-bit counterparts.&quot; This is a very deceptive statement. Take 100 initialization states and train a FP16 vs a FP32 network, and you&#x27;ll find FP32 will have an accuracy advantage. It&#x27;s certainly possible to conclude this if a small sample of networks are trained. This paper goes on to state, &quot;Lowering the precision of real numbers used for the neural network’s weights to fixed-point, as shown in [11], leads to a significant decrease in accuracy.&quot;, while later concluding, &quot;we have shown that pure 16-bit networks can perform on par with, if not better than, mixed-precision and 32-bit networks in various image classification tasks.&quot; The results certainly do, but that doesn&#x27;t really give an accurate evaluation of what&#x27;s really going on here. A FP64 network can fall into a local minima and be outperformed by a PF16 network, but is it correct to say the FP16 network is better. I&#x27;m getting a lot of mixed signals.<p>I feel like, &quot;significant implications&quot; is quite a stretch.<p>A few concerns: Besides figure 3, other results do not provide side-by-side test vs validation accuracy to attempt demonstrate the network is not overfit, and the only mention of normalization was the custom batch normalization operation.<p>This may more be a rant about the current state of ML, but in a perfect world, we wouldn&#x27;t use GPUs&#x2F;would enforce deterministic calculations, results would be replicable, we&#x27;d train hundreds if not thousands of networks to draw conclusions from, we&#x27;d better understand how to visualize network accuracies and overfitting, and all datasets would be free of bias and accurately generalize the problem attempting to be modelled. We can dream.