hi all,
I am trying to create a most simple 1 linear layer network to fit a linear regression. Just to help myself better understand how Pytorch works. However, I encountered a strange issue with the model training.
in my model’s init() method, I have to add a manual initialization step(shown below) to have the model quickly converge to my regression function. (the weight value 2, 3 are random number, I could put any value here and the model will still converge)
self.layer1.weight = torch.nn.Parameter(torch.Tensor([2, 3]))

Without this line, the model never converge, the training loss just randomly oscillates in the range of hundreds of thousands. With this line, it quickly decreases to near 1.
I have postulated that it is because default initial weight parameters were too small if I do not initialize them to be far away from zero. Then I changed the initial values and found out the convergence always work as long as I have this line, the exact value I set does not matter. Could someone explain what is going on behind the scene here? Thanks.
My entire script:
import torch
import numpy as np

class Net(torch.nn.Module):
    
    def __init__(self, input_dim, output_dim):
        super(Net, self).__init__()
        self.layer1 = torch.nn.Linear(input_dim, output_dim, bias=False)
        self.layer1.weight = torch.nn.Parameter(torch.Tensor([2, 3]))
        
    def forward(self, x):
        x = self.layer1(x)
        x.squeeze()
        return x
        
# generate data using the linear regression setup y = 5 * x1 + 3 * x2

sample_size = 10000
input_dim = 2
output_dim = 1
epoch = 30
bs = 100

data = np.random.randn(sample_size, 3)
data[:, :2] = data[:, :2] * 100
# add a normal noise term
data[:, 2] = 5 * data[:, 0] + 3 * data[:, 1] + np.random.randn(sample_size)
data = torch.Tensor(data)
train_x = data[:, :input_dim]
train_y = data[:, input_dim]

net = Net(input_dim, output_dim)
net.zero_grad()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.RMSprop(net.parameters(), lr=.01)

for i in range(epoch):
    
    batch = 0
    while batch * bs < train_x.shape[0]:
        
        batch_x = train_x[batch * bs : (batch + 1) * bs, :]
        batch_y = train_y[batch * bs : (batch + 1) * bs]
        
        pred_y = net.forward(batch_x)
        loss = criterion(pred_y, batch_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if batch % 100 == 0:
            #print(f"{i} {batch} {loss}")
            print(net.layer1.weight)
        batch += 1