Hi Barthelemy!



 barthelemymp:

I’m was making a neural network to to try a few option of pytorch, and then when I tried it on the classical Breast Cancer dataset my algo was just stuck. I simplified to the maximum my model, but accurracy seem to be blocked.


Let me speculate that you are building a binary classification
model – i.e., some sample either is or is not breast cancer, based
on some kind of input features taken from a breast-tissue sample.


my model:
class Modela(torch.nn.Module):
    def __init__(self):
        super(Modela, self).__init__()
        self.l1 = torch.nn.Linear(30,16)
        self.l2 = torch.nn.Linear(16,4)
        self.l3 = torch.nn.Linear(4,2)
        
        self.sigmoid = torch.nn.Sigmoid()
        self.relu = torch.nn.ReLU()



(Your last layer, self.l3, has two outputs, suggesting that this
may be binary.)


    def forward(self, x):
        out1 = self.relu(self.l1(x)) 
        out2 = self.relu(self.l2(out1))
        y_pred = self.sigmoid(self.l3(out2))
        return y_pred



I think your main problem is that you are calling self.sigmoid()
on the output of your last layer.  Given that (from your code further
down) you are using nn.CrossEntropyLoss as your loss function,
you should be using the outputs of l3 as your y_pred and pass
them directly to nn.CrossEntropyLoss.  This is because (maybe
a little bit inconsistently with its name) CrossEntropyLoss expects
logits (numbers running from -infinity to  infinity) as its inputs, but
your sigmoid maps the logits coming out of l3 to (0, 1).
So try getting rid of the sigmoid, and see if that helps.
A couple more comments:
If this is a binary classification problem, you should consider
having your last layer have a single output, and using
nn.BCEWithLogitsLoss as your loss function.  This won’t
change the actual math or results of your network, but will
make it a little simpler and more efficient.
Also, it’s not clear that having three layers / two hidden layers
makes your network better.  It could make it worse or harder
to train.  You might try a single hidden layer with something
like 16 hidden neurons (as you have in your first hidden layer).
So your code fragment (also getting rid of the sigmoid and
changing to the binary version instead of the general multi-class
version) might look something like:
class Modela(torch.nn.Module):
    def __init__(self):
        super(Modela, self).__init__()
        self.l1 = torch.nn.Linear(30,16)
        self.l2 = torch.nn.Linear(16,1)

        self.relu = torch.nn.ReLU()
        
    def forward(self, x):
        hidden1 = self.relu(self.l1(x)) 
        y_pred = self.l2(hidden1)
        return y_pred

# and to match the single output of l2
criterion = torch.nn.BCEWithLogitsLoss()

Whether a single hidden layer will work better than two will be
problem dependent, but it’s likely to, so it’s worth trying.  Also,
the best number of hidden neurons will be problem dependent,
so you should experiment some, but, given that you have 30
inputs (and one output), something like 16 hidden neurons is
a reasonable place to start.
Best.
K. Frank