Yes, I understand. The throughput calculation is over many updates, so the slow down is real.
And for the comparisons, I keep the hardware all same. Even though P40 doesn’t have TensorCores, I did see speedups with APEX O2 in two of the three configurations above. And the one in which I installed Pytorch with pip install torch==1.6.0+cu101, the speed up is gone.