Summary: FP16 with apex or AMP will only give you some memory savings with a reasonably high batch size.
Additionally, under mixed precision when possible, it's important that the batch size is a multiple of 8 to efficiently use tensor cores.
Note that in some situations the speed up can be as big as 5x when using mixed precision. e.g. we have observed that while using Megatron-Deepspeed.
Some amazing tutorials to read on mixed precision:
- @sgugger wrote a great explanation of mixed precision here
- Aleksey Bilogur's A developer-friendly guide to mixed precision training with PyTorch
You can also see a variety of benchmarks on fp16 vs other precisions:
RTX-3090 and
A100.
fp16 caching
pytorch autocast which performs AMP include a caching feature, which speed things up by caching fp16-converted values. Here is the full description from this comment:
