If you will use standard pytorch blocks (contained in nn module) - they will. If you create tensors by yourself that should be updated by autograd you need to wrap them inside nn.Parameter. In any case you can check if parameter is actually updated by checking value of your_custom_parameter.grad after doing loss.backward(). If it will be non-zero that means it will be updated by the optimizer.