I have a custom module with two sets of nn.Parameters():
class CustomLayer(nn.Module):
    def __init__(self):
        self.param1 = nn.Parameter(torch.rand())
        self.param2 = nn.Parameter(torch.rand())

later when I optimize with optimizer.step():
layer = CustomLayer()
optimizer = torch.optim.SGD(layer.parameters(), lr=0.5)
optimizer.step()

it doesn’t do param = param - 0.5*param.grad it instead calculates: param = param - param.grad