Body: <p>I trained a complex cnn but when I look at the training and validation loss, it seems that the model is overfitting. But due to limitation on resources I can't increase the datasize. Is there any other way to overcome the overfitting. I have attached the code for training and the model below.</p>
  <pre><code>class CV_CNN_Net(nn.Module):
      def __init__(self,device):
          super(CV_CNN_Net, self).__init__()
          self.device = device
          self.nn = nn.Sequential(
                                  ComplexConv2d(1, 128, kernel_size = (3, 3),stride = (2, 2),padding=(1,1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  nn.Flatten(),
                                  ComplexLinear(2048,1024),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexDropout(p=0.3,device = self.device),
                                  ComplexLinear(1024, 512),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexDropout(p=0.3,device = self.device),
                                  ComplexLinear(512, 73),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
                                  ComplexDropout(p=0.3, device=self.device),
                                  )
          self.FNN = nn.Sequential(
              nn.Linear(146, 73, bias=False),
              nn.Sigmoid()
          )
      def forward(self, data_train):
          Outputs = self.nn(data_train)
          Outputs = torch.cat([Outputs.real,Outputs.imag],dim=1)
          Outputs = self.FNN(Outputs)
          return Outputs
  </code></pre>
  <p>The training:</p>
  <pre><code>def one_epoch(model,optimizer,lr,criterion,L1_Regularizer,dataloader,device,epoch_now,epoch_all,is_train,is_L1_Regularizer):
      if is_train==True:
          model.train()
      else:
          model.eval()
      mean_loss = torch.zeros(1).to(device)
      dataloader = tqdm(dataloader, desc=f'Epoch {epoch_now + 1}/{epoch_all}', file=sys.stdout)
      for step, data in enumerate(dataloader):
          x_batch, y_batch = data
          ouputs_train = model(x_batch.to(device))
          loss = criterion(ouputs_train, y_batch.to(device).to(torch.float32))
          if is_L1_Regularizer == True:
              for model_param_name, model_param_value in model.named_parameters():
                  if model_param_name in ['FNN.0.weight']:
                      loss = L1_Regularizer.regularized_param(param_weights=model_param_value,
                                                                    reg_loss_function=loss)
          mean_loss = (mean_loss * step + loss.detach()) / (step + 1)  # update mean losses
          if is_train==True:
              optimizer.zero_grad()
              loss.backward()
              optimizer.step()
              #dataloader.desc = &quot;[Training epoch {}] mean loss:{}&quot;.format(epoch_now, round(mean_loss.item(), 6))+' lr:{}'.format(lr)
          else:
              dataloader.desc = &quot;[Val epoch {}] mean loss:{}&quot;.format(epoch_now,round(mean_loss.item(), 6)) + ' lr:{}'.format(lr)
      return mean_loss.item()
  def Pre_Train():
      xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Pre_train, Validation_set_size,
                                                       Cov_Matrix_type_Pre_train)
      criterion = nn.BCELoss().to(device)
      L1_Regularizer = L1Regularizer(model, lambda_reg=5e-5)
      optimizer = optim.Adam(model.parameters(), lr=1e-4, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
      scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
      Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Pre_train, shuffle=True)
      Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Pre_train, shuffle=True)
      Val_loss_min = 99999.0
      for epoch in range(epochs_Pre_train):
  
          Train_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                                 Train_dataloader,device,epoch,epochs_Pre_train,is_train=True,is_L1_Regularizer=True)
  
  
          Val_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                               Val_dataloader,device,epoch,epochs_Pre_train,is_train=False,is_L1_Regularizer=True)
  
          scheduler.step(Val_loss)
  
          writer_Pre_train.add_scalar(&quot;Train_loss&quot;, Train_loss, epoch)
          writer_Pre_train.add_scalar(&quot;Val_loss&quot;, Val_loss, epoch)
          writer_Pre_train.add_scalar(&quot;lr&quot;, optimizer.param_groups[0]['lr'], epoch)
  
          if (epoch + 1) % 10 == 0:
              torch.save(model, filename_save_model_Pre_train + &quot;/model_{}.pth&quot;.format(epoch + 1))
          if Val_loss_min &gt; Val_loss:
              Val_loss_min = Val_loss
              torch.save(model, filename_save_model_Pre_train + &quot;/model_best.pth&quot;)
  
  
  def Fine_tuning():
      xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Fine_tuning, Validation_set_size,
                                               Cov_Matrix_type_Fine_tuning)
      # initialize Fine_tuning model with  paremeters of Pre_train model
      checkpoint = torch.load(filename_save_model_Pre_train + &quot;/model_200.pth&quot;, map_location=device)
      model.load_state_dict(checkpoint.state_dict())
  
      criterion = nn.BCELoss().to(device)
      L1_Regularizer = L1Regularizer(model, lambda_reg=1e-5)
      optimizer = optim.Adam(model.parameters(), lr=5e-5, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
      scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
      Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Fine_tuning, shuffle=True)
      Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Fine_tuning, shuffle=True)
  
      Val_loss_min = 99999.0
      train_losses, val_losses = [], []
      
      for epoch in range(epochs_Fine_tuning):
  
          Train_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                                 Train_dataloader, device, epoch,epochs_Fine_tuning, is_train=True, is_L1_Regularizer=True)
  
  
          Val_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                               Val_dataloader, device,  epoch,epochs_Fine_tuning, is_train=False, is_L1_Regularizer=True)
  
          scheduler.step(Val_loss)
          train_losses.append(Train_loss)
          val_losses.append(Val_loss)
  
          writer_Fine_tuning.add_scalar(&quot;Train_loss&quot;, Train_loss, epoch)
          writer_Fine_tuning.add_scalar(&quot;Val_loss&quot;, Val_loss, epoch)
          writer_Fine_tuning.add_scalar(&quot;lr&quot;, optimizer.param_groups[0]['lr'], epoch)
  
          if (epoch + 1) % 10 == 0:
              torch.save(model, filename_save_model_Fine_tuning + &quot;/model_{}.pth&quot;.format(epoch + 1))
          if Val_loss_min &gt; Val_loss:
              Val_loss_min = Val_loss
              torch.save(model, filename_save_model_Fine_tuning + &quot;/model_best.pth&quot;)
      plot_losses(train_losses, val_losses, title=&quot;Fine-Tuning Loss Over Epochs&quot;)
  if __name__ == &quot;__main__&quot;:
      if os.path.exists(filename_save_model_Pre_train) is False:
          os.makedirs(filename_save_model_Pre_train)
      if os.path.exists(filename_save_model_Fine_tuning) is False:
          os.makedirs(filename_save_model_Fine_tuning)
      Pre_Train()
      Fine_tuning()
  </code></pre>
  <p>Currently the training loss looks as follows:<a href="https://i.sstatic.net/AJUF6Np8.png" rel="nofollow noreferrer">[Loss]</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-12T14:02:16.030Z
Comments.Text: I was thinking if I train the network multiple times on different sets of data, will it help in generalization?
Comments.UserDisplayName: ananya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-12T19:54:22.407Z
Comments.Text: With BatchNorm and Dropout already, have you tried k-fold cross validation and image augmentation trick via random rotating/flipping/cropping/zooming/color shifting, etc?
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-13T08:28:28.830Z
Comments.Text: @cinch, due to constraint on training resources, I can't use huge amount of data as it will take alot of time to train. So, I didn't increase the dataset using data augmentation. Still, I have used around 40,000 training samples and it took almost 20hrs to train.
Comments.UserDisplayName: ananya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-13T08:40:48.347Z
Comments.Text: Your result graph clearly shows overfit to your training set. So most likely a cross validation with your limited data resources is a good way to tune some of your parameters and hyperparameters like number of cnn nodes/layers to reduce your validation error.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-13T10:34:07.970Z
Comments.Text: @cinch, without k-fold cross validation, for 300 epochs the training is taking around 20hrs, so even if include k = 5 fold, it will take around 3 days. Given that my model isn't that deep, do you think it should take so much time to train?
Comments.UserDisplayName: ananya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-13T22:31:54.390Z
Comments.Text: It’s possible since you’re using complex valued cov2d layers, try use real valued simple layers and increase dropout to 0.5, or reduce filter size to reduce your model’s complexity.
Comments.UserDisplayName: cinch
Tags: <convolutional-neural-networks><deep-neural-networks><overfitting>
Title: How to overcome overfitting when you can't increase the training data due to constraint in time and resources?
UserDisplayName: ananya
ThreadText: # How to overcome overfitting when you can't increase the training data due to constraint in time and resources?
  
  (Asked by: username_0 on 2024-11-12)
  
  I trained a complex cnn but when I look at the training and validation loss, it seems that the model is overfitting. But due to limitation on resources I can't increase the datasize. Is there any other way to overcome the overfitting. I have attached the code for training and the model below.
  class CV_CNN_Net(nn.Module):
      def __init__(self,device):
          super(CV_CNN_Net, self).__init__()
          self.device = device
          self.nn = nn.Sequential(
                                  ComplexConv2d(1, 128, kernel_size = (3, 3),stride = (2, 2),padding=(1,1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                  ComplexBatchNorm2d(128),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  nn.Flatten(),
                                  ComplexLinear(2048,1024),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexDropout(p=0.3,device = self.device),
                                  ComplexLinear(1024, 512),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
  
                                  ComplexDropout(p=0.3,device = self.device),
                                  ComplexLinear(512, 73),
                                  # ComplexReLU(),
                                  C_CSELU(),
                                  # ZReLU(),
                                  ComplexDropout(p=0.3, device=self.device),
                                  )
          self.FNN = nn.Sequential(
              nn.Linear(146, 73, bias=False),
              nn.Sigmoid()
          )
      def forward(self, data_train):
          Outputs = self.nn(data_train)
          Outputs = torch.cat([Outputs.real,Outputs.imag],dim=1)
          Outputs = self.FNN(Outputs)
          return Outputs
  
  The training:
  def one_epoch(model,optimizer,lr,criterion,L1_Regularizer,dataloader,device,epoch_now,epoch_all,is_train,is_L1_Regularizer):
      if is_train==True:
          model.train()
      else:
          model.eval()
      mean_loss = torch.zeros(1).to(device)
      dataloader = tqdm(dataloader, desc=f'Epoch {epoch_now + 1}/{epoch_all}', file=sys.stdout)
      for step, data in enumerate(dataloader):
          x_batch, y_batch = data
          ouputs_train = model(x_batch.to(device))
          loss = criterion(ouputs_train, y_batch.to(device).to(torch.float32))
          if is_L1_Regularizer == True:
              for model_param_name, model_param_value in model.named_parameters():
                  if model_param_name in ['FNN.0.weight']:
                      loss = L1_Regularizer.regularized_param(param_weights=model_param_value,
                                                                    reg_loss_function=loss)
          mean_loss = (mean_loss * step + loss.detach()) / (step + 1)  # update mean losses
          if is_train==True:
              optimizer.zero_grad()
              loss.backward()
              optimizer.step()
              #dataloader.desc = "[Training epoch {}] mean loss:{}".format(epoch_now, round(mean_loss.item(), 6))+' lr:{}'.format(lr)
          else:
              dataloader.desc = "[Val epoch {}] mean loss:{}".format(epoch_now,round(mean_loss.item(), 6)) + ' lr:{}'.format(lr)
      return mean_loss.item()
  def Pre_Train():
      xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Pre_train, Validation_set_size,
                                                       Cov_Matrix_type_Pre_train)
      criterion = nn.BCELoss().to(device)
      L1_Regularizer = L1Regularizer(model, lambda_reg=5e-5)
      optimizer = optim.Adam(model.parameters(), lr=1e-4, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
      scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
      Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Pre_train, shuffle=True)
      Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Pre_train, shuffle=True)
      Val_loss_min = 99999.0
      for epoch in range(epochs_Pre_train):
  
          Train_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                                 Train_dataloader,device,epoch,epochs_Pre_train,is_train=True,is_L1_Regularizer=True)
  
  
          Val_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                               Val_dataloader,device,epoch,epochs_Pre_train,is_train=False,is_L1_Regularizer=True)
  
          scheduler.step(Val_loss)
  
          writer_Pre_train.add_scalar("Train_loss", Train_loss, epoch)
          writer_Pre_train.add_scalar("Val_loss", Val_loss, epoch)
          writer_Pre_train.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)
  
          if (epoch + 1) % 10 == 0:
              torch.save(model, filename_save_model_Pre_train + "/model_{}.pth".format(epoch + 1))
          if Val_loss_min > Val_loss:
              Val_loss_min = Val_loss
              torch.save(model, filename_save_model_Pre_train + "/model_best.pth")
  
  
  def Fine_tuning():
      xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Fine_tuning, Validation_set_size,
                                               Cov_Matrix_type_Fine_tuning)
      # initialize Fine_tuning model with  paremeters of Pre_train model
      checkpoint = torch.load(filename_save_model_Pre_train + "/model_200.pth", map_location=device)
      model.load_state_dict(checkpoint.state_dict())
  
      criterion = nn.BCELoss().to(device)
      L1_Regularizer = L1Regularizer(model, lambda_reg=1e-5)
      optimizer = optim.Adam(model.parameters(), lr=5e-5, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
      scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
      Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Fine_tuning, shuffle=True)
      Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Fine_tuning, shuffle=True)
  
      Val_loss_min = 99999.0
      train_losses, val_losses = [], []
      
      for epoch in range(epochs_Fine_tuning):
  
          Train_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                                 Train_dataloader, device, epoch,epochs_Fine_tuning, is_train=True, is_L1_Regularizer=True)
  
  
          Val_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                               Val_dataloader, device,  epoch,epochs_Fine_tuning, is_train=False, is_L1_Regularizer=True)
  
          scheduler.step(Val_loss)
          train_losses.append(Train_loss)
          val_losses.append(Val_loss)
  
          writer_Fine_tuning.add_scalar("Train_loss", Train_loss, epoch)
          writer_Fine_tuning.add_scalar("Val_loss", Val_loss, epoch)
          writer_Fine_tuning.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)
  
          if (epoch + 1) % 10 == 0:
              torch.save(model, filename_save_model_Fine_tuning + "/model_{}.pth".format(epoch + 1))
          if Val_loss_min > Val_loss:
              Val_loss_min = Val_loss
              torch.save(model, filename_save_model_Fine_tuning + "/model_best.pth")
      plot_losses(train_losses, val_losses, title="Fine-Tuning Loss Over Epochs")
  if __name__ == "__main__":
      if os.path.exists(filename_save_model_Pre_train) is False:
          os.makedirs(filename_save_model_Pre_train)
      if os.path.exists(filename_save_model_Fine_tuning) is False:
          os.makedirs(filename_save_model_Fine_tuning)
      Pre_Train()
      Fine_tuning()
  
  Currently the training loss looks as follows:[Loss]
  
  **Tags:** convolutional-neural-networks, deep-neural-networks, overfitting
  > **Comments:**
  > **username_0**: I was thinking if I train the network multiple times on different sets of data, will it help in generalization?  
  > **username_1**: With BatchNorm and Dropout already, have you tried k-fold cross validation and image augmentation trick via random rotating/flipping/cropping/zooming/color shifting, etc?  
  > **username_0**: @cinch, due to constraint on training resources, I can't use huge amount of data as it will take alot of time to train. So, I didn't increase the dataset using data augmentation. Still, I have used around 40,000 training samples and it took almost 20hrs to train.  
  > **username_1**: Your result graph clearly shows overfit to your training set. So most likely a cross validation with your limited data resources is a good way to tune some of your parameters and hyperparameters like number of cnn nodes/layers to reduce your validation error.  
  > **username_0**: @cinch, without k-fold cross validation, for 300 epochs the training is taking around 20hrs, so even if include k = 5 fold, it will take around 3 days. Given that my model isn't that deep, do you think it should take so much time to train?  
  > **username_1**: It’s possible since you’re using complex valued cov2d layers, try use real valued simple layers and increase dropout to 0.5, or reduce filter size to reduce your model’s complexity.  
  
  ---
  
  ## 0 answers
  
  