Hi,
After converting my program into mixed-precision using amp, the forward time gets shorter while the backward time gets longer when I record running time by “import time”.
Then I use “torch.profiler”  to record the running time.
image1704×1143 56 KB
However, it seems that the result of torch.profiler meets my expectation: cuda.time becomes around 1/4 of the fp32.
Here my question is: what is the cuda.time means and why the backpropagation time becomes longer in mix precision?
mixed-precision:
import time :
Entire Epoch: [1]	Train: [0]	SUM: 1828.379	DT: 13.270	FW: 71.987	BK: 1711.248	CM: 31.873	
torch profiler:
Self CPU time total: 56.883s
CUDA time total: 893.112s
FP32:
import time :
Entire Epoch: [3]	Train: [0]	SUM: 1584.260	DT: 15.714	FW: 368.550	BK: 1119.231	CM: 80.766
torch profiler:
Self CPU time total: 105.049s
CUDA time total: 3021.355s
Detailed Logs of my program are  listed below:
http://49.234.107.127:81/index.php/s/qa3Yjo8WJwNZjCS (mixed precision)
http://49.234.107.127:81/index.php/s/y8SpyfiM3d5SZp7
My model uses conv3d and I run my code on Tesla V100.
Many thanks.