The default dtype is float32, which works faster in CUDA than float64.
Usually you don’t need the precision and range of FP64, and can stick to FP32.
However, Tensorcores in newer NVIDIA GPUs can accelerate FP16 operations.
Since the value range and precision might be too narrow for some models using this number format, we developed NVIDIA/apex containing mixed precision recipes and are currently working on upstreaming it into PyTorch in this issue 2.