Hi shinra!



 shinra:

    for param in model.parameters():
        param=param-param.grad*learning_rate

Loss is not reducing…


This is more of a python question than something specific to pytorch.
In:
    for param in model.parameters():

param is python variable which means that it’s a reference to
something.  In each iteration through the for loop it is set to refer
to one of the model.parameters().
But when you run the body of the loop:
        param=param-param.grad*learning_rate

you first create a new python tensor,
param-param.grad*learning_rate,
and then set the variable param to refer to that new tensor:
        param=param-param.grad*learning_rate

This changes what param refers to, but doesn’t change the value
of the parameter in model.parameters() to which param used to
refer.
Consider the pure python:
a = [1, 2, 3]
b = a
b = [4, 5, 6]

You can easily verify that after running this b refers to a list that
has the value [4, 5, 6], but that a still refers to the list with value
[1, 2, 3].
This is the same thing that is happening with your variable param
and the actual parameter in model.parameters() that param
temporarily referred to.
(The most convenient way to do what you want is to use
opt = torch.optim.SGD (...), and call opt.step().
Don’t forget to “zero your grads” (e.g., opt.zero_grad())
before calling loss.backward() in each iteration of your
training loop.)
Best.
K. Frank