And indeed, if you want a completely new optimizer rather than control gradients by hand, you should inherit from torch.optim.Optimizer and implement the step method. (remember to wrap it with @torch.no_grad())