@ptrblck
As we know cudnn.benchmark=True is generally recommended  for speed ups so turning it off would slow down the training considerably.
Also, even if FP16 uses less memory in the run, we see that even with cudnn disabled the reduction of memory is very minimal (cause peak memory consumptions would cause the OOM) so unless and until these peak memory consumptions go down, GPUs will run out of memory. What should be the mitigation strategy in this case.