When enable the benchmark, it speedup a bit but still 2x slower since 1080ti don’t have Tensor core. But I wanted to achieve the effect of the Apex O1 mode : constant speed and low memory footprint. Now the autocast() is like the apex O2 mode, which is also 2x slower on the 1080ti.