According to
https://tigress-web.princeton.edu/~jdh4/PyTorchPerformanceTuningGuide_GTC2021.pdf 71 (Slide 6)
it seems preferable to call model.zero_grad() for performance reasons (pytorch>=1.7).
EDIT: thanks to ptrblck, I should have put model.zero_grad(set_to_none=True).