For anybody who arrives here through a google search -
This is a paper by Nvidia which sheds more light on training in FP16.
https://arxiv.org/abs/1710.03740 242
Also they have been nice to provide implemented code -

  
      
      GitHub
  
  
    

NVIDIA/apex 89

A PyTorch Extension:  Tools for easy mixed precision and distributed training in Pytorch - NVIDIA/apex