We’ve developed a lightweight, open-source set of Pytorch tools to enable easier, more numerically stable mixed precision training:  https://github.com/nvidia/apex 1.1k.  Mixed precision means that the majority of the network uses FP16 arithmetic (reducing memory storage/bandwidth demands and enabling Tensor Cores for gemms and convolutions), while a small subset of operations are executed in FP32 for improved stability.
Highlights include:

Amp, a tool that executes all numerically safe Torch functions in FP16, while automatically casting potentially unstable operations to FP32.  Amp also automatically implements dynamic loss scaling.  Amp is designed to offer maximum numerical stability, and most of the speed benefits of pure FP16 training.
FP16_Optimizer, an optimizer wrapper that automatically implements FP32 master weights for parameter updates, as well as static or dynamic loss scaling.  FP16_Optimizer is designed to be minimally invasive (it doesn’t change the execution of Torch operations) and offer almost all the speed of pure FP16 training with significantly improved numerical stability.
apex.parallel.DistributedDataParallel, a distributed module wrapper that achieves high performance by overlapping computation with communication during backward().  Apex DistributedDataParallel is useful for both pure FP32 as well as mixed precision training.

Full API documentation can be found here 291.