<p>Your problem type is more suited for classification than regression as you want to assign a class to the second largest number and not predict a value. Check out loss functions like CrossEntropyLoss rather than MSE to solve this type of problem.</p>

CC BY-SA 4.0
2024-08-27T09:38:00.990Z
2024-08-27T09:38:00.990Z
renderbender
<p>To practice what I learned about PyTorch, I gave myself the following problem:</p>
<blockquote>
<p>Create a model that given a vector, predicts what the 2nd largest number in it is.</p>
</blockquote>
<p>For example, <code>model([ 0.3, 0.4. 0.9 0.7 ])</code> should return <code>[ 0.0 0.0 0.0 1.0 ]</code>. I am pretty new to neural networks so I tried a feed forward network. However, no matter how large I make the dataset or how many training iterations I do I don't see my loss function decreasing and my model at the end gives pretty much random results. I feel like this should be a pretty easy problem to solve with a neural network, so what am I doing wrong here? Is a feed forward design not appropriate here?</p>
<pre><code>import torch as t
import torch.nn as nn
import torch.optim as optim

# Example: input [ 0.3, 0.4. 0.9 0.7 ] -&gt; [ 0.0 0.0 0.0 1.0 ]
training_set = []
for i in range(10000):
    data_point = t.rand(4)
    target = sorted(data_point)[-2]
    label = t.tensor([ 1. if v == target else 0. for v in data_point ])
    training_set.append([data_point, label])

class Model(nn.Module):
    def __init__(self, hidden=100):
        super(Model, self).__init__()
        self.lin1 = nn.Linear(4, hidden)
        self.sig = nn.Sigmoid()
        self.lin2 = nn.Linear(hidden, 4)

    # Simple feed forward network
    def forward(self, input):
        return self.lin2(self.sig(self.lin1(input)))

model = Model()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters())

for epoch in range(3):
    running_loss = 0.0

    for i, (input, label) in enumerate(training_set):
        optimizer.zero_grad()
        pred = model(input) # Forward pass
        loss = criterion(pred, label) # Calculate loss
        loss.backward() # Backward pass
        optimizer.step()
        
        # Print statistics
        running_loss += loss.item()
        step = 500
        if i % step == 0:
            print(epoch, i, running_loss / step)
            running_loss = 0.0

test = t.rand(4)
print('Input', test)
print('Output', model(test))
</code></pre>

CC BY-SA 4.0
2024-08-30T07:15:40.800Z
I'd say that probably this task is not the best as it is not a true "machine learning" problem. There will also be problems with your model, for instance it is not invariant to the order of the input, despite the fact that $[x, y]$ and $[y, x]$ should yield the same answer. You would be better just generating some artificial data (or using one of the many datasets used in ML courses) to perform some regression/classification.
David
<neural-networks><pytorch><feedforward-neural-networks>
Loss Function not Decreasing
Dan
# Loss Function not Decreasing

(Asked by: username_0 on 2024-08-26)

To practice what I learned about PyTorch, I gave myself the following problem:

Create a model that given a vector, predicts what the 2nd largest number in it is.

For example, model([ 0.3, 0.4. 0.9 0.7 ]) should return [ 0.0 0.0 0.0 1.0 ]. I am pretty new to neural networks so I tried a feed forward network. However, no matter how large I make the dataset or how many training iterations I do I don't see my loss function decreasing and my model at the end gives pretty much random results. I feel like this should be a pretty easy problem to solve with a neural network, so what am I doing wrong here? Is a feed forward design not appropriate here?
import torch as t
import torch.nn as nn
import torch.optim as optim

# Example: input [ 0.3, 0.4. 0.9 0.7 ] -> [ 0.0 0.0 0.0 1.0 ]
training_set = []
for i in range(10000):
    data_point = t.rand(4)
    target = sorted(data_point)[-2]
    label = t.tensor([ 1. if v == target else 0. for v in data_point ])
    training_set.append([data_point, label])

class Model(nn.Module):
    def __init__(self, hidden=100):
        super(Model, self).__init__()
        self.lin1 = nn.Linear(4, hidden)
        self.sig = nn.Sigmoid()
        self.lin2 = nn.Linear(hidden, 4)

    # Simple feed forward network
    def forward(self, input):
        return self.lin2(self.sig(self.lin1(input)))

model = Model()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters())

for epoch in range(3):
    running_loss = 0.0

    for i, (input, label) in enumerate(training_set):
        optimizer.zero_grad()
        pred = model(input) # Forward pass
        loss = criterion(pred, label) # Calculate loss
        loss.backward() # Backward pass
        optimizer.step()
        
        # Print statistics
        running_loss += loss.item()
        step = 500
        if i % step == 0:
            print(epoch, i, running_loss / step)
            running_loss = 0.0

test = t.rand(4)
print('Input', test)
print('Output', model(test))

**Tags:** neural-networks, pytorch, feedforward-neural-networks

> **Comments:**
> **username_1**: I'd say that probably this task is not the best as it is not a true "machine learning" problem. There will also be problems with your model, for instance it is not invariant to the order of the input, despite the fact that $[x, y]$ and $[y, x]$ should yield the same answer. You would be better just generating some artificial data (or using one of the many datasets used in ML courses) to perform some regression/classification.  

---

## 1 answers

### Answer by username_2 on 2024-08-27. Score: 0

Your problem type is more suited for classification than regression as you want to assign a class to the second largest number and not predict a value. Check out loss functions like CrossEntropyLoss rather than MSE to solve this type of problem.


---

