title: Strange behavior of linear regression in PyTorch
body: <p>I am facing a peculiar problem and I was wondering if there is an explanation. I am trying to run a linear regression problem and test different optimization methods and two of them have a strange outcome when comparing to each other. I build a data set that satisfies y=2x+5 and I add a random noise to that.</p>
  
  <pre><code> xtrain=np.range(0,50,1).reshape(50,1)
   ytrain=2*train+5+np.random.normal(0,2,(50,1))
   opt1=torch.optim.SGD(model.parameters(),lr=1e-5,momentum=0.8))
   opt2=torch.optim.Rprop(model.parameters(),lr=1e-5)
   F_loss=F.mse_loss
   from torch.utils.data import TensorDataset,DataLoader
   train_d=TensorDataset(xtrain,ytrain)     
   train=DataLoader(train_d,50,shuffle=True)     
   model1=nn.Linear(1,1)
   loss=F_loss(model1(xtrain),ytrain)
  
   def fit(nepoch, model1, F_loss, opt):
       for epoch in range(nepoch):
           for i,j in train:
           predict = model1(i)
           loss = F_loss(predict, j)
           loss.backward()
           opt.step()
           opt.zero_grad()
  </code></pre>
  
  <p>When i compare the results of the following commands:</p>
  
  <pre><code>        fit(500000, model1, F_loss, opt1)
          fit(500000, model1, F_loss, opt2)
  </code></pre>
  
  <p>In the last epoch for opt1:loss=2.86,weight=2.02,bias=4.46<br/>
   In the last epoch for opt2:loss=3.47,weight=2.02,bias=4.68</p>
  
  <p>These results do not make sense to me, shouldn't <code>opt2</code> have a smaller loss than <code>opt1</code> since the weight and bias it finds is closer to the real value? <code>opt2</code>'s method finds weights and biases to be closer to the real value (they are respectively 2 and 5). Am i doing something wrong? </p>