Thanks @ptrblck for your help. I manage to get access to a pair of RTX2080Ti. Here the speed of torch.amp is similar to apex. I guess torch.amp is optimized to use TensorCores to boost the speed, given that GTX GPUs do not have them, the computation time is slower. Could you confirm this?