Answers.Body: <p>Your problem type is more suited for classification than regression as you want to assign a class to the second largest number and not predict a value. Check out loss functions like CrossEntropyLoss rather than MSE to solve this type of problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-27T09:38:00.990Z
Answers.LastActivityDate: 2024-08-27T09:38:00.990Z
Answers.UserDisplayName: renderbender
Body: <p>To practice what I learned about PyTorch, I gave myself the following problem:</p>
  <blockquote>
  <p>Create a model that given a vector, predicts what the 2nd largest number in it is.</p>
  </blockquote>
  <p>For example, <code>model([ 0.3, 0.4. 0.9 0.7 ])</code> should return <code>[ 0.0 0.0 0.0 1.0 ]</code>. I am pretty new to neural networks so I tried a feed forward network. However, no matter how large I make the dataset or how many training iterations I do I don't see my loss function decreasing and my model at the end gives pretty much random results. I feel like this should be a pretty easy problem to solve with a neural network, so what am I doing wrong here? Is a feed forward design not appropriate here?</p>
  <pre><code>import torch as t
  import torch.nn as nn
  import torch.optim as optim
  
  # Example: input [ 0.3, 0.4. 0.9 0.7 ] -&gt; [ 0.0 0.0 0.0 1.0 ]
  training_set = []
  for i in range(10000):
      data_point = t.rand(4)
      target = sorted(data_point)[-2]
      label = t.tensor([ 1. if v == target else 0. for v in data_point ])
      training_set.append([data_point, label])
  
  class Model(nn.Module):
      def __init__(self, hidden=100):
          super(Model, self).__init__()
          self.lin1 = nn.Linear(4, hidden)
          self.sig = nn.Sigmoid()
          self.lin2 = nn.Linear(hidden, 4)
  
      # Simple feed forward network
      def forward(self, input):
          return self.lin2(self.sig(self.lin1(input)))
  
  model = Model()
  criterion = nn.MSELoss()
  optimizer = optim.SGD(model.parameters())
  
  for epoch in range(3):
      running_loss = 0.0
  
      for i, (input, label) in enumerate(training_set):
          optimizer.zero_grad()
          pred = model(input) # Forward pass
          loss = criterion(pred, label) # Calculate loss
          loss.backward() # Backward pass
          optimizer.step()
          
          # Print statistics
          running_loss += loss.item()
          step = 500
          if i % step == 0:
              print(epoch, i, running_loss / step)
              running_loss = 0.0
  
  test = t.rand(4)
  print('Input', test)
  print('Output', model(test))
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-30T07:15:40.800Z
Comments.Text: I'd say that probably this task is not the best as it is not a true "machine learning" problem. There will also be problems with your model, for instance it is not invariant to the order of the input, despite the fact that $[x, y]$ and $[y, x]$ should yield the same answer. You would be better just generating some artificial data (or using one of the many datasets used in ML courses) to perform some regression/classification.
Comments.UserDisplayName: David
Tags: <neural-networks><pytorch><feedforward-neural-networks>
Title: Loss Function not Decreasing
UserDisplayName: Dan
ThreadText: # Loss Function not Decreasing
  
  (Asked by: username_0 on 2024-08-26)
  
  To practice what I learned about PyTorch, I gave myself the following problem:
  
  Create a model that given a vector, predicts what the 2nd largest number in it is.
  
  For example, model([ 0.3, 0.4. 0.9 0.7 ]) should return [ 0.0 0.0 0.0 1.0 ]. I am pretty new to neural networks so I tried a feed forward network. However, no matter how large I make the dataset or how many training iterations I do I don't see my loss function decreasing and my model at the end gives pretty much random results. I feel like this should be a pretty easy problem to solve with a neural network, so what am I doing wrong here? Is a feed forward design not appropriate here?
  import torch as t
  import torch.nn as nn
  import torch.optim as optim
  
  # Example: input [ 0.3, 0.4. 0.9 0.7 ] -> [ 0.0 0.0 0.0 1.0 ]
  training_set = []
  for i in range(10000):
      data_point = t.rand(4)
      target = sorted(data_point)[-2]
      label = t.tensor([ 1. if v == target else 0. for v in data_point ])
      training_set.append([data_point, label])
  
  class Model(nn.Module):
      def __init__(self, hidden=100):
          super(Model, self).__init__()
          self.lin1 = nn.Linear(4, hidden)
          self.sig = nn.Sigmoid()
          self.lin2 = nn.Linear(hidden, 4)
  
      # Simple feed forward network
      def forward(self, input):
          return self.lin2(self.sig(self.lin1(input)))
  
  model = Model()
  criterion = nn.MSELoss()
  optimizer = optim.SGD(model.parameters())
  
  for epoch in range(3):
      running_loss = 0.0
  
      for i, (input, label) in enumerate(training_set):
          optimizer.zero_grad()
          pred = model(input) # Forward pass
          loss = criterion(pred, label) # Calculate loss
          loss.backward() # Backward pass
          optimizer.step()
          
          # Print statistics
          running_loss += loss.item()
          step = 500
          if i % step == 0:
              print(epoch, i, running_loss / step)
              running_loss = 0.0
  
  test = t.rand(4)
  print('Input', test)
  print('Output', model(test))
  
  **Tags:** neural-networks, pytorch, feedforward-neural-networks
  > **Comments:**
  > **username_1**: I'd say that probably this task is not the best as it is not a true "machine learning" problem. There will also be problems with your model, for instance it is not invariant to the order of the input, despite the fact that $[x, y]$ and $[y, x]$ should yield the same answer. You would be better just generating some artificial data (or using one of the many datasets used in ML courses) to perform some regression/classification.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-08-27. Score: 0
  
  Your problem type is more suited for classification than regression as you want to assign a class to the second largest number and not predict a value. Check out loss functions like CrossEntropyLoss rather than MSE to solve this type of problem.
  
  
  ---
  
  