Setting gradients to None will not necessarily slow down training as we have some optimizations in place to avoid re-allocating the gradient buffer and just re-use the intermediary buffers from the backward.
But that does not work all the time, depending on many factors.
So you will have to try out for your model to know the exact impact.