Hi Saif!



 Saif_Ali_Khan:

weight vector is completely different than the original.


I’ve implemented a version of what I believe you are doing and
Linear.weight converges to the weight vector used to generate
the training data.
Here is the script:
import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

a = torch.randn (3)
b = torch.randn (1)

model = torch.nn.Linear (3, 1)

nOpt = 100
nPrint = 10
opt = torch.optim.SGD (model.parameters(), lr = 0.1)
loss_fn = torch.nn.MSELoss()

for  i in range (nOpt):
    x = torch.randn (3)
    t = a @ x + b
    p = model (x)
    loss = loss_fn (p, t)
    if  i % (nOpt / nPrint) == 0  or  (i + 1) == nOpt:
        print ('loss =', loss)
    opt.zero_grad()
    loss.backward()
    opt.step()

print ('a =', a)
print ('model.weight =', model.weight.data)
print ('b =', b)
print ('model.bias =', model.bias.data)

And here is its output:
1.7.1
loss = tensor(81.5180, grad_fn=<MseLossBackward>)
loss = tensor(0.8514, grad_fn=<MseLossBackward>)
loss = tensor(0.0013, grad_fn=<MseLossBackward>)
loss = tensor(5.5018e-06, grad_fn=<MseLossBackward>)
loss = tensor(1.4914e-05, grad_fn=<MseLossBackward>)
loss = tensor(7.4943e-07, grad_fn=<MseLossBackward>)
loss = tensor(2.5289e-07, grad_fn=<MseLossBackward>)
loss = tensor(1.3833e-07, grad_fn=<MseLossBackward>)
loss = tensor(8.2538e-10, grad_fn=<MseLossBackward>)
loss = tensor(2.2737e-13, grad_fn=<MseLossBackward>)
loss = tensor(1.4211e-12, grad_fn=<MseLossBackward>)
a = tensor([ 2.2871,  0.6413, -0.8615])
model.weight = tensor([[ 2.2871,  0.6413, -0.8615]])
b = tensor([-0.3649])
model.bias = tensor([-0.3649])

Best.
K. Frank