Hi all!
I’m trying to implement a simple neural network to address a binary classification problem on an imbalanced dataset (21% positive vs 79% negative).
I can’t find any configuration (hyper parameters or model architecture) that make the training/val loss decrease: the loss is highly unstable around the 0.6 value (I assume it’s near the log(2)) so I wondering what could be the problem. I performed a dataset cleaning process in which I reduced the features dimensionality from 45 to 8: however, the correlation of each feature to the label is between 0.13 and 0.21 and this worry me.
To solve the imbalanced problem I oversample the positive data during the training in such a way to have 50-50 pos-neg ratio in each batch.
Actually, this is my model architecture:
class Network(nn.Module):
 
    def __init__(self,in_ch,out_ch):
        super().__init__()
        # Inputs to hidden layer linear transformation
        self.fc1 = nn.Linear(in_ch, 16)
        self.bn1 = nn.BatchNorm1d(16)
        self.fc2 = nn.Linear(16, 32)
        self.bn2 = nn.BatchNorm1d(32)
        self.fc3 = nn.Linear(32, 64)
        self.bn3 = nn.BatchNorm1d(64)
        self.fc4 = nn.Linear(64, 32)
        self.fc5 = nn.Linear(32, out_ch)
        

    def forward(self, x):
        x = F.leaky_relu(self.bn1(self.fc1(x)))
        x = F.leaky_relu(self.bn2(self.fc2(x)))
        x = F.leaky_relu(self.bn3(self.fc3(x)))
        x = F.leaky_relu(self.fc4(x))
        return self.fc5(x)

I’ve already tried different activations functions (relu and tanh) and also to remove batch normalization.
I use a simple SGD optimizer.
Do you have any suggestion on what could be the problem?