willy:

So TensorCores are used by default in PyTorch already?


On Ampere GPUs, yes, since the TF32 type 1 is used.


cudnn.benchmark = True will profile different algorithms for each convolution and select the fastest one. The algorithm selection will be stored in a cache for each new input shape. The performance tuning guide 1 explains it in more detail. It’s also not specific to AMP.


Yes, potentially an additional speedup can be seen, which might not be as large as pure FP32 training vs. AMP.


No, all CUDA versions are supported. The pip wheels and conda binaries are shipped with CUDA10.2 and 11.1 at the moment, but you can build from source with any toolkit >=10.0.


The channels-last memory format could yield a speedup when using AMP (refer to the performance guide). If you are seeing a slowdown, please post the model definition and input shapes, which would reproduce the issue.