Hey I found the bug. I wasn’t clearing the lists in which I stored the batches properly!
In my data_loader:
  def data_loader(folder, batch_size):
      """ Our dataset is very unbalanced, so I am forcing our data loader
          to load the same amount of positive and negative samples.
      """
      patients_list = []  # Mistake!
      labels_list = []  # Mistake!
      while True:
          for _ in range(batch_size):
              label = random.choice(['0', '1'])
              patient_ids = os.listdir(os.path.join(folder, label))
              patient_id = random.choice(patient_ids)
              patient_path = os.path.join(folder, label, patient_id)
              patients_list.append(torch.load(patient_path))
              label = torch.Tensor([int(label)])
              labels_list.append(label)

          batch_variable = Variable(torch.stack(patients_list),
                                    requires_grad=False)
          batch_labels = torch.squeeze(Variable(torch.stack(labels_list),
                                                requires_grad=False))

          yield batch_variable, batch_labels.long().cuda()

patients_list = [] and labels_list = [] should be inside the while loop! Else I would be appending to the list infinitely.
Thanks for taking the time to helping me debug this!