Thanks a lot for your reply and implementation. I copied your code and you’re right, it does work. But when i merge your code with mine (The only difference is that i’m training whole x which is of shape [125 x 3]). In the end, i print out rmse and last 15 output i.e y[110:] (you can notice they’re the same, but still weights are different. This time even bias is also different)
x_path = Path('../datasets/') / 'synthetic_datasets/1/X.npy'
x = np.load(x_path)
x = torch.from_numpy(x)
x = x.type(torch.FloatTensor)

_ = torch.manual_seed (2021)
a = torch.randn (3)
b = torch.randn (1)
model = torch.nn.Linear (3, 1)
t = a @ x.T + b
t = torch.reshape(t, (len(t),1))
print(t.shape)
nOpt = 10000
optimizer = torch.optim.SGD (model.parameters(), lr = 0.01)
loss_fn = torch.nn.MSELoss()

for  i in range (nOpt):
    y_pred = model(x)
    optimizer.zero_grad()
    y_pred = model(x)
    
    loss = loss_fn(y_pred, t)
    loss.backward()
    optimizer.step()
prediction(model, x, t, 'regression')
print ('a =', a)
print ('b =', b)
for param in model_new.parameters():
    print(param)

Output:

torch.Size([125, 3])
1.7.1
torch.Size([125, 1])
tensor([[10.0661],
[ 9.2046],
[ 8.3431],
[ 7.4816],
[ 6.6201],
[10.7075],
[ 9.8460],
[ 8.9845],
[ 8.1230],
[ 7.2614],
[11.3488],
[10.4873],
[ 9.6258],
[ 8.7643],
[ 7.9028]]) [[10.0661125]
[ 9.204607 ]
[ 8.3431015]
[ 7.481596 ]
[ 6.6200905]
[10.7074585]
[ 9.845953 ]
[ 8.9844475]
[ 8.122942 ]
[ 7.2614365]
[11.348804 ]
[10.487299 ]
[ 9.625793 ]
[ 8.764288 ]
[ 7.902782 ]]
Rmse: 5.683192e-06
a = tensor([ 2.2871,  0.6413, -0.8615])
b = tensor([-0.3649])
Parameter containing:
tensor([[ 4.0516, -9.3466,  6.8186]], requires_grad=True)
Parameter containing:
tensor([1.1346], requires_grad=True)