There’s no magic in the Module implementation, it’s only a simple container and is not involved in optimization process. The problem with your module-less examples is that you never zero the weight gradient weight.grad.data.zero_(), while in the module example you call optimizer.zero_grad() that does it for you. Once I added that they’re both converging to the solution.