Answers.Body: <p>There are a few things you could do to improve this NN, but are probably worth covering in different questions.</p>
  
  <p>Your main problem though is that you forgot to reset the gradient after each training batch. You need to call <code>optim.zero_grad()</code> in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep accumulating inside the model's training data (sometimes you want this effect if you are adding gradients from multiple sources, that's why PyTorch is not clearing them automatically for you).</p>
  
  <p>In addition, a learning rate of 0.5 is very high for the Adam optimiser - it is very common to leave it at the default value because Adam is an adaptive optimiser that will adjust step sizes depending on gradients seen so far.</p>
  
  <p>Here is a working version of your code:</p>
  
  <pre><code>import torch
  import torch.nn as nn
  import torch.nn.functional as F
  
  class Net(nn.Module):
      def __init__(self):
  
          super(Net,self).__init__();
  
          self.fc1 = nn.Linear(in_features=2,out_features=3)
          self.fc2 = nn.Linear(in_features=3,out_features=6)
          self.out = nn.Linear(in_features=6,out_features=1)
  
      def forward(self,x):
          x = self.fc1(x)
          x = torch.relu(x)
          x = self.fc2(x)
          x = torch.relu(x)
          x = self.out(x)
          return x
  
  data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
  values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)
  
  lossfun = torch.nn.MSELoss()
  model=Net()
  optim = torch.optim.Adam(model.parameters(),lr=0.001)
  
  for epoch in range(20000):
      optim.zero_grad()
      y_pred=model.forward(data)
      loss = lossfun(y_pred,values)
      if (epoch % 1000 == 0):
          print(loss.item())
      loss.backward()
      optim.step()
  </code></pre>
  
  <p>This version can be tweaked to quite easily reach 0 loss for your data set.</p>
  
  <p>This has not in really learned how to multiply two values. The approximation to multiplying will be very weak as there is very little data. However, playing with some very basic data and a simple NN is a first step towards understanding details like this . . .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-27T16:46:32.800Z
Answers.LastActivityDate: 2019-03-27T16:53:38.730Z
Answers.LastEditDate: 2019-03-27T16:53:38.730Z
Answers.UserDisplayName: Neil Slater
Body: <p>As my first AI model I have decided to make an AI model to predict multiplication of two numbers EX - [2,4] = [8]. I wrote the following code, but the loss is very high, around thousands, and it's very inaccurate. How do I make it more accurate?</p>
  
  <pre><code>import torch
  import torch.nn as nn
  import torch.nn.functional as F
  
  data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
  values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)
  lossfun = torch.nn.MSELoss()
  model=Net()
  optim = torch.optim.Adam(model.parameters(),lr=0.5)
  
  class Net(nn.Module):
      def __init__(self):
  
          super(Net,self).__init__();
  
          self.fc1 = nn.Linear(in_features=2,out_features=3)
          self.fc2 = nn.Linear(in_features=3,out_features=6)
          self.out = nn.Linear(in_features=6,out_features=1)
  
      def forward(self,x):
          x = self.fc1(x)
          x = F.relu(x)
          x = self.fc2(x)
          x = F.relu(x)
          x = self.out(x)
          return x
  for epoch in range(1000):
  
      y_pred=model.forward(data)
  
      loss = lossfun(y_pred,values)
  
      print(loss.item())
  
      loss.backward()
  
      optim.step()
  </code></pre>
  
  <p>Note: I am a newbie in AI and ML.</p>
  
Tags: <objective-functions><pytorch>
Title: Heavy loss and inaccurate answer in pytorch
UserDisplayName: Mohammad Ummair
ThreadText: # Heavy loss and inaccurate answer in pytorch
  
  (Asked by: username_0 on 2019-03-27)
  
  As my first AI model I have decided to make an AI model to predict multiplication of two numbers EX - [2,4] = [8]. I wrote the following code, but the loss is very high, around thousands, and it's very inaccurate. How do I make it more accurate?
  
  ```
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  
  data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
  values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)
  lossfun = torch.nn.MSELoss()
  model=Net()
  optim = torch.optim.Adam(model.parameters(),lr=0.5)
  
  class Net(nn.Module):
      def __init__(self):
  
          super(Net,self).__init__();
  
          self.fc1 = nn.Linear(in_features=2,out_features=3)
          self.fc2 = nn.Linear(in_features=3,out_features=6)
          self.out = nn.Linear(in_features=6,out_features=1)
  
      def forward(self,x):
          x = self.fc1(x)
          x = F.relu(x)
          x = self.fc2(x)
          x = F.relu(x)
          x = self.out(x)
          return x
  for epoch in range(1000):
  
      y_pred=model.forward(data)
  
      loss = lossfun(y_pred,values)
  
      print(loss.item())
  
      loss.backward()
  
      optim.step()
  
  ```
  
  Note: I am a newbie in AI and ML.
  
  **Tags:** objective-functions, pytorch
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-27. Score: 2 | Accepted answer
  
  There are a few things you could do to improve this NN, but are probably worth covering in different questions.
  
  Your main problem though is that you forgot to reset the gradient after each training batch. You need to call `optim.zero_grad()` in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep accumulating inside the model's training data (sometimes you want this effect if you are adding gradients from multiple sources, that's why PyTorch is not clearing them automatically for you).
  
  In addition, a learning rate of 0.5 is very high for the Adam optimiser - it is very common to leave it at the default value because Adam is an adaptive optimiser that will adjust step sizes depending on gradients seen so far.
  
  Here is a working version of your code:
  
  ```
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  
  class Net(nn.Module):
      def __init__(self):
  
          super(Net,self).__init__();
  
          self.fc1 = nn.Linear(in_features=2,out_features=3)
          self.fc2 = nn.Linear(in_features=3,out_features=6)
          self.out = nn.Linear(in_features=6,out_features=1)
  
      def forward(self,x):
          x = self.fc1(x)
          x = torch.relu(x)
          x = self.fc2(x)
          x = torch.relu(x)
          x = self.out(x)
          return x
  
  data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
  values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)
  
  lossfun = torch.nn.MSELoss()
  model=Net()
  optim = torch.optim.Adam(model.parameters(),lr=0.001)
  
  for epoch in range(20000):
      optim.zero_grad()
      y_pred=model.forward(data)
      loss = lossfun(y_pred,values)
      if (epoch % 1000 == 0):
          print(loss.item())
      loss.backward()
      optim.step()
  
  ```
  
  This version can be tweaked to quite easily reach 0 loss for your data set.
  
  This has not in really learned how to multiply two values. The approximation to multiplying will be very weak as there is very little data. However, playing with some very basic data and a simple NN is a first step towards understanding details like this . . .
  
  
  ---
  
  