<p>I trained a complex cnn but when I look at the training and validation loss, it seems that the model is overfitting. But due to limitation on resources I can't increase the datasize. Is there any other way to overcome the overfitting. I have attached the code for training and the model below.</p>
<pre><code>class CV_CNN_Net(nn.Module):
    def __init__(self,device):
        super(CV_CNN_Net, self).__init__()
        self.device = device
        self.nn = nn.Sequential(
                                ComplexConv2d(1, 128, kernel_size = (3, 3),stride = (2, 2),padding=(1,1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                nn.Flatten(),
                                ComplexLinear(2048,1024),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(1024, 512),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(512, 73),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),
                                ComplexDropout(p=0.3, device=self.device),
                                )
        self.FNN = nn.Sequential(
            nn.Linear(146, 73, bias=False),
            nn.Sigmoid()
        )
    def forward(self, data_train):
        Outputs = self.nn(data_train)
        Outputs = torch.cat([Outputs.real,Outputs.imag],dim=1)
        Outputs = self.FNN(Outputs)
        return Outputs
</code></pre>
<p>The training:</p>
<pre><code>def one_epoch(model,optimizer,lr,criterion,L1_Regularizer,dataloader,device,epoch_now,epoch_all,is_train,is_L1_Regularizer):
    if is_train==True:
        model.train()
    else:
        model.eval()
    mean_loss = torch.zeros(1).to(device)
    dataloader = tqdm(dataloader, desc=f'Epoch {epoch_now + 1}/{epoch_all}', file=sys.stdout)
    for step, data in enumerate(dataloader):
        x_batch, y_batch = data
        ouputs_train = model(x_batch.to(device))
        loss = criterion(ouputs_train, y_batch.to(device).to(torch.float32))
        if is_L1_Regularizer == True:
            for model_param_name, model_param_value in model.named_parameters():
                if model_param_name in ['FNN.0.weight']:
                    loss = L1_Regularizer.regularized_param(param_weights=model_param_value,
                                                                  reg_loss_function=loss)
        mean_loss = (mean_loss * step + loss.detach()) / (step + 1)  # update mean losses
        if is_train==True:
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            #dataloader.desc = &quot;[Training epoch {}] mean loss:{}&quot;.format(epoch_now, round(mean_loss.item(), 6))+' lr:{}'.format(lr)
        else:
            dataloader.desc = &quot;[Val epoch {}] mean loss:{}&quot;.format(epoch_now,round(mean_loss.item(), 6)) + ' lr:{}'.format(lr)
    return mean_loss.item()
def Pre_Train():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Pre_train, Validation_set_size,
                                                     Cov_Matrix_type_Pre_train)
    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=5e-5)
    optimizer = optim.Adam(model.parameters(), lr=1e-4, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Pre_train, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Pre_train, shuffle=True)
    Val_loss_min = 99999.0
    for epoch in range(epochs_Pre_train):

        Train_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                               Train_dataloader,device,epoch,epochs_Pre_train,is_train=True,is_L1_Regularizer=True)


        Val_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                             Val_dataloader,device,epoch,epochs_Pre_train,is_train=False,is_L1_Regularizer=True)

        scheduler.step(Val_loss)

        writer_Pre_train.add_scalar(&quot;Train_loss&quot;, Train_loss, epoch)
        writer_Pre_train.add_scalar(&quot;Val_loss&quot;, Val_loss, epoch)
        writer_Pre_train.add_scalar(&quot;lr&quot;, optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Pre_train + &quot;/model_{}.pth&quot;.format(epoch + 1))
        if Val_loss_min &gt; Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Pre_train + &quot;/model_best.pth&quot;)


def Fine_tuning():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Fine_tuning, Validation_set_size,
                                             Cov_Matrix_type_Fine_tuning)
    # initialize Fine_tuning model with  paremeters of Pre_train model
    checkpoint = torch.load(filename_save_model_Pre_train + &quot;/model_200.pth&quot;, map_location=device)
    model.load_state_dict(checkpoint.state_dict())

    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=1e-5)
    optimizer = optim.Adam(model.parameters(), lr=5e-5, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Fine_tuning, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Fine_tuning, shuffle=True)

    Val_loss_min = 99999.0
    train_losses, val_losses = [], []
    
    for epoch in range(epochs_Fine_tuning):

        Train_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                               Train_dataloader, device, epoch,epochs_Fine_tuning, is_train=True, is_L1_Regularizer=True)


        Val_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                             Val_dataloader, device,  epoch,epochs_Fine_tuning, is_train=False, is_L1_Regularizer=True)

        scheduler.step(Val_loss)
        train_losses.append(Train_loss)
        val_losses.append(Val_loss)

        writer_Fine_tuning.add_scalar(&quot;Train_loss&quot;, Train_loss, epoch)
        writer_Fine_tuning.add_scalar(&quot;Val_loss&quot;, Val_loss, epoch)
        writer_Fine_tuning.add_scalar(&quot;lr&quot;, optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Fine_tuning + &quot;/model_{}.pth&quot;.format(epoch + 1))
        if Val_loss_min &gt; Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Fine_tuning + &quot;/model_best.pth&quot;)
    plot_losses(train_losses, val_losses, title=&quot;Fine-Tuning Loss Over Epochs&quot;)
if __name__ == &quot;__main__&quot;:
    if os.path.exists(filename_save_model_Pre_train) is False:
        os.makedirs(filename_save_model_Pre_train)
    if os.path.exists(filename_save_model_Fine_tuning) is False:
        os.makedirs(filename_save_model_Fine_tuning)
    Pre_Train()
    Fine_tuning()
</code></pre>
<p>Currently the training loss looks as follows:<a href="https://i.sstatic.net/AJUF6Np8.png" rel="nofollow noreferrer">[Loss]</a></p>

CC BY-SA 4.0
2024-11-12T14:02:16.030Z
I was thinking if I train the network multiple times on different sets of data, will it help in generalization?
ananya
CC BY-SA 4.0
2024-11-12T19:54:22.407Z
With BatchNorm and Dropout already, have you tried k-fold cross validation and image augmentation trick via random rotating/flipping/cropping/zooming/color shifting, etc?
cinch
CC BY-SA 4.0
2024-11-13T08:28:28.830Z
@cinch, due to constraint on training resources, I can't use huge amount of data as it will take alot of time to train. So, I didn't increase the dataset using data augmentation. Still, I have used around 40,000 training samples and it took almost 20hrs to train.
ananya
CC BY-SA 4.0
2024-11-13T08:40:48.347Z
Your result graph clearly shows overfit to your training set. So most likely a cross validation with your limited data resources is a good way to tune some of your parameters and hyperparameters like number of cnn nodes/layers to reduce your validation error.
cinch
CC BY-SA 4.0
2024-11-13T10:34:07.970Z
@cinch, without k-fold cross validation, for 300 epochs the training is taking around 20hrs, so even if include k = 5 fold, it will take around 3 days. Given that my model isn't that deep, do you think it should take so much time to train?
ananya
CC BY-SA 4.0
2024-11-13T22:31:54.390Z
It’s possible since you’re using complex valued cov2d layers, try use real valued simple layers and increase dropout to 0.5, or reduce filter size to reduce your model’s complexity.
cinch
<convolutional-neural-networks><deep-neural-networks><overfitting>
How to overcome overfitting when you can't increase the training data due to constraint in time and resources?
ananya
# How to overcome overfitting when you can't increase the training data due to constraint in time and resources?

(Asked by: username_0 on 2024-11-12)

I trained a complex cnn but when I look at the training and validation loss, it seems that the model is overfitting. But due to limitation on resources I can't increase the datasize. Is there any other way to overcome the overfitting. I have attached the code for training and the model below.
class CV_CNN_Net(nn.Module):
    def __init__(self,device):
        super(CV_CNN_Net, self).__init__()
        self.device = device
        self.nn = nn.Sequential(
                                ComplexConv2d(1, 128, kernel_size = (3, 3),stride = (2, 2),padding=(1,1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                nn.Flatten(),
                                ComplexLinear(2048,1024),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(1024, 512),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(512, 73),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),
                                ComplexDropout(p=0.3, device=self.device),
                                )
        self.FNN = nn.Sequential(
            nn.Linear(146, 73, bias=False),
            nn.Sigmoid()
        )
    def forward(self, data_train):
        Outputs = self.nn(data_train)
        Outputs = torch.cat([Outputs.real,Outputs.imag],dim=1)
        Outputs = self.FNN(Outputs)
        return Outputs

The training:
def one_epoch(model,optimizer,lr,criterion,L1_Regularizer,dataloader,device,epoch_now,epoch_all,is_train,is_L1_Regularizer):
    if is_train==True:
        model.train()
    else:
        model.eval()
    mean_loss = torch.zeros(1).to(device)
    dataloader = tqdm(dataloader, desc=f'Epoch {epoch_now + 1}/{epoch_all}', file=sys.stdout)
    for step, data in enumerate(dataloader):
        x_batch, y_batch = data
        ouputs_train = model(x_batch.to(device))
        loss = criterion(ouputs_train, y_batch.to(device).to(torch.float32))
        if is_L1_Regularizer == True:
            for model_param_name, model_param_value in model.named_parameters():
                if model_param_name in ['FNN.0.weight']:
                    loss = L1_Regularizer.regularized_param(param_weights=model_param_value,
                                                                  reg_loss_function=loss)
        mean_loss = (mean_loss * step + loss.detach()) / (step + 1)  # update mean losses
        if is_train==True:
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            #dataloader.desc = "[Training epoch {}] mean loss:{}".format(epoch_now, round(mean_loss.item(), 6))+' lr:{}'.format(lr)
        else:
            dataloader.desc = "[Val epoch {}] mean loss:{}".format(epoch_now,round(mean_loss.item(), 6)) + ' lr:{}'.format(lr)
    return mean_loss.item()
def Pre_Train():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Pre_train, Validation_set_size,
                                                     Cov_Matrix_type_Pre_train)
    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=5e-5)
    optimizer = optim.Adam(model.parameters(), lr=1e-4, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Pre_train, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Pre_train, shuffle=True)
    Val_loss_min = 99999.0
    for epoch in range(epochs_Pre_train):

        Train_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                               Train_dataloader,device,epoch,epochs_Pre_train,is_train=True,is_L1_Regularizer=True)


        Val_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                             Val_dataloader,device,epoch,epochs_Pre_train,is_train=False,is_L1_Regularizer=True)

        scheduler.step(Val_loss)

        writer_Pre_train.add_scalar("Train_loss", Train_loss, epoch)
        writer_Pre_train.add_scalar("Val_loss", Val_loss, epoch)
        writer_Pre_train.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Pre_train + "/model_{}.pth".format(epoch + 1))
        if Val_loss_min > Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Pre_train + "/model_best.pth")


def Fine_tuning():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Fine_tuning, Validation_set_size,
                                             Cov_Matrix_type_Fine_tuning)
    # initialize Fine_tuning model with  paremeters of Pre_train model
    checkpoint = torch.load(filename_save_model_Pre_train + "/model_200.pth", map_location=device)
    model.load_state_dict(checkpoint.state_dict())

    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=1e-5)
    optimizer = optim.Adam(model.parameters(), lr=5e-5, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Fine_tuning, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Fine_tuning, shuffle=True)

    Val_loss_min = 99999.0
    train_losses, val_losses = [], []
    
    for epoch in range(epochs_Fine_tuning):

        Train_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                               Train_dataloader, device, epoch,epochs_Fine_tuning, is_train=True, is_L1_Regularizer=True)


        Val_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                             Val_dataloader, device,  epoch,epochs_Fine_tuning, is_train=False, is_L1_Regularizer=True)

        scheduler.step(Val_loss)
        train_losses.append(Train_loss)
        val_losses.append(Val_loss)

        writer_Fine_tuning.add_scalar("Train_loss", Train_loss, epoch)
        writer_Fine_tuning.add_scalar("Val_loss", Val_loss, epoch)
        writer_Fine_tuning.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Fine_tuning + "/model_{}.pth".format(epoch + 1))
        if Val_loss_min > Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Fine_tuning + "/model_best.pth")
    plot_losses(train_losses, val_losses, title="Fine-Tuning Loss Over Epochs")
if __name__ == "__main__":
    if os.path.exists(filename_save_model_Pre_train) is False:
        os.makedirs(filename_save_model_Pre_train)
    if os.path.exists(filename_save_model_Fine_tuning) is False:
        os.makedirs(filename_save_model_Fine_tuning)
    Pre_Train()
    Fine_tuning()

Currently the training loss looks as follows:[Loss]

**Tags:** convolutional-neural-networks, deep-neural-networks, overfitting

> **Comments:**
> **username_0**: I was thinking if I train the network multiple times on different sets of data, will it help in generalization?  
> **username_1**: With BatchNorm and Dropout already, have you tried k-fold cross validation and image augmentation trick via random rotating/flipping/cropping/zooming/color shifting, etc?  
> **username_0**: @username_1, due to constraint on training resources, I can't use huge amount of data as it will take alot of time to train. So, I didn't increase the dataset using data augmentation. Still, I have used around 40,000 training samples and it took almost 20hrs to train.  
> **username_1**: Your result graph clearly shows overfit to your training set. So most likely a cross validation with your limited data resources is a good way to tune some of your parameters and hyperparameters like number of cnn nodes/layers to reduce your validation error.  
> **username_0**: @username_1, without k-fold cross validation, for 300 epochs the training is taking around 20hrs, so even if include k = 5 fold, it will take around 3 days. Given that my model isn't that deep, do you think it should take so much time to train?  
> **username_1**: It’s possible since you’re using complex valued cov2d layers, try use real valued simple layers and increase dropout to 0.5, or reduce filter size to reduce your model’s complexity.  

---

## 0 answers

