So it depends on cudnn’s update …
Because my current model (pytorch version of stylegan2 [Karras]) uses a lot of custom conv layers(must process convolutions by groups), O1/O2 setting makes no big difference with O0 mode. Hope Nvidia updates faster kernels soon.
I also found some pytorch functions (e.g. torch.repeat, torch.cat) is slow in O1 (functions are patched as fp16) mode. In O2, these functions are faster as O0(fp32) but overall backpropagation is slower than O1.