Hello Ra!



 Ra-V:

I’m trying to train a network for simple classification task on randomly generated dataset with a bit imbalanced classes (59 observations of class 0 and 140 of class 1), and I can’t seem to teach the NN to distinguish between them, it always just simply predicts all the classes to class 1. I tried to use different weighting of classes, but it didn’t help.
Here is my code:
class NeuralNet(nn.Module):
  def __init__(self, n_input, n_hidden1, n_hidden2, output_shape):
    super().__init__()
    self.hidden2 = nn.Linear(n_hidden1, n_hidden2)
    self.hidden = nn.Linear(n_input, n_hidden1)
    self.outer = nn.Linear(n_hidden1, output_shape)
    self.sigmoid = nn.Sigmoid()
    self.softmax = nn.Softmax(dim=1)
...
model = NeuralNet(4, 5, 5, 2)
...
loss_function = F.binary_cross_entropy



Your passing the wrong information / shape to binary_cross_entropy.
binary_cross_entropy expects one prediction value per sample, to
be understood as the probability of that sample being in class “1”.
(It expects a single target value per sample, as well.)  You construct
your last linear layer to have two outputs – you should have one.
When you switch to a single output, you will need to switch from
Softmax to Sigmoid in your final layer (after the last linear layer).
However, for numerical reasons, you will be better off using
binary_cross_entropy_with_logits, fed directly by your last
linear layer, and not have a Sigmoid.
In order to reweight your unbalanced training data (You might
not need to.) you will need to use the per-sample weight
argument to binary_cross_entropy_with_logits.  That is,
if a specific sample in your batch is class “0” you give it the
reweighting weight you want for class “0”; if it’s class “1”, you
give it the class “1” weight.
One other note:  It doesn’t matter in your specific case because
you have n_hidden1 = n_hidden2 (= 5), but you should have
n_hidden2 inputs to your last linear layer:
self.outer = nn.Linear(n_hidden2, output_shape)
(You can treat this as a multiclass classification problem – with just
two classes – and have an output layer with two outputs, but then
you would need to use cross_entropy.  I would expect it to be
modestly more efficient to train a network that has one output with
binary_cross_entropy.)
Good luck!
K. Frank