In pytorch you have 2 optimizer methods which are usually used:
op.zero_grad(), op.step() and loss.backward()
if you just dont set gradients to zero you will accumulate them. step is nothing but upgrade model parameters