Sorry for the late reply.
I’ve reran the transformer model with a source build using the latest CUDA version and see a speedup on the V100s you are using as: AMP: 16.79ms, FP32: 20.02ms, so I assume cublas might have updated their heuristics and pick faster (TC) kernels.
@aminsaied I wouldn’t expect to see a large difference between AMP and FP32 on Maxwell GPUs.
@jsleep I don’t agree, as apex/amp is deprecated and used the same autocast list in the then recommended O1 mode.