Just a question out of curiosity, in pytorch the current gradients of the parameters is automatically added to the previous ones, therefore we have to set all the gradients to zero after each update.
So is it true that by doing that it does cause performance regression since we are doing two unwanted stuff repeatedly - 1. Set the previous gradients to zero and - 2. Then add the current gradients to the previous gradients.
Also is there a way to stop the default addition of gradients.