I’m working on a binary classification neural network with severe class imbalance (33130 zeros and 415 ones). While training my network, i test the accuracy for each epoch and it starts with 40%, reaches to 85% after 30 epochs. I’m using F1 score, recall and precision to measure the accuracy. the results vary a lot when i change the batch size of the dataloader.
when i feed the validation dataloader to the model however, accuracy plummets to 0%.
I don’t have the slightest idea what would be causing this, so ill try to include only essential information in this post while giving as much details as possible.
I started by splitting the dataset into features and labels, I then imputed the features, scaled it and reduced it using my SimpleImputer, StandardScaler and PCA objects
I then split the data using train_test_split
x_train, x_val, y_train, y_val = train_test_split(reduced_features, 
                                                  labels, 
                                                  test_size = 0.33, 
                                                  random_state = 0)

I used WeightedRandomSampler to deal with class imbalance while following the guide in this article: PyTorch [Basics] — Sampling 4
target_list = torch.tensor(y_train.values)
target_list = target_list[torch.randperm(len(target_list))]

class_count = y_train.value_counts()
class_weights = 1./torch.tensor(class_count, dtype=torch.float)

class_weights_all = class_weights[target_list]

weighted_sampler = WeightedRandomSampler(
    weights=class_weights_all,
    num_samples=len(class_weights_all),
    replacement=True
)

I created the following class to pass an object containing features and labels to the dataloader:
class Data_for_loader(Dataset):
    
    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data
        
    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]
        
    def __len__ (self):
        return len(self.X_data)


train_data = Data_for_loader(torch.FloatTensor(x_train.values), 
                             torch.FloatTensor(y_train.values))

validation_data = Data_for_loader(torch.FloatTensor(x_val.values),
                            torch.FloatTensor(y_val.values))

I then created the dataloader objects:
train_loader = DataLoader(dataset = train_data, shuffle=False, batch_size = 150, sampler=weighted_sampler)
validation_loader = DataLoader(dataset = validation_data, batch_size= 1)

since this is a binary classification problem I’m using BCEWithLogitsLoss as my criterion
I’m using Adam as my optimizer
I created the following class for my model:
class BinaryClassifier(nn.Module):
    def __init__(self):
        super(BinaryClassifier, self).__init__()

        self.h1 = nn.Linear(140, 90) 
        self.h2 = nn.Linear(90, 50)
        self.output = nn.Linear(50, 1) 
        
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.2)
        self.batchnorm1 = nn.BatchNorm1d(90)
        self.batchnorm2 = nn.BatchNorm1d(50)
        
    def forward(self, x):
        x = self.h1(x)
        x = self.relu(x)
        x = self.batchnorm1(x)
        
        x = self.h2(x)
        x = self.relu(x)
        x = self.batchnorm2(x)

        x = self.dropout(x)
        x = self.output(x)
        
        return x

I used the following code to train the model:
model.train()

for e in range(epochs):
    
    epoch_loss = 0   
    
    recall_total_score = 0  
    precision_total_score = 0
    f1_total_score = 0
    
    for inputs, labels in train_loader:
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        
        output = model.forward(inputs)
        loss = criterion(output, labels.unsqueeze(1))
        
        #rounding output for accuracy measure
        output = torch.round(torch.sigmoid(output))
        output = output.cpu().detach().numpy()
        labels = labels.cpu().unsqueeze(1)
        
        #measuring accuracy
        recall_total_score += recall_score(labels, output)
        f1_total_score += f1_score(labels, output)
        precision_total_score += precision_score(labels, output)
        
        
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()

finaly i used the following code to evaluate the model:
y_pred = []
y_true = []

model.eval()
with torch.no_grad():
    for inputs, labels in validation_loader:
        inputs = inputs.to(device)
        
        output = model.forward(inputs)
        output = torch.round(torch.sigmoid(output))

        y_pred.extend(output.cpu().numpy())
        y_true.extend(labels.cpu().numpy())

f1 = f1_score(y_true, y_pred)

I’m relatively new to Pytorch, my only guess is that the model is overfitting, even tho I’m using dropout in my model, or i’m somehow making a mistake while dealing with the validation set.
Thanks for your help!