Hello, I’m trying to train a pre-trained Faster R-CNN in Google Collab Pro. My function for the training is OK and everything works very well.
But I tried to launch a validation step using the exact same function except that i don’t use the backpropagation when I computed the loss.
But when I skip the backpropagation step, I get an error of CUDA MEMORY, whilst when I apply the backpropagation (during the training) I don’t have any error of CUDA MEMORY even without using the   torch.cuda.empty_cache(). This is my code :
def train_one_epoch(model, optimizer, data_loader, device, validation):    
    model.train()
    for i, values in enumerate(data_loader):
        images, targets = values
        images = list(image.to(device) for image in images)
        targets = [{'boxes' : t['boxes'] , 'labels' : t['labels'].to(device)} for t in targets ]
        loss_dict = model(images, targets)
        losses = sum(loss for loss in loss_dict.values())
        optimizer.zero_grad()

        if validation == False :               
            losses.backward()
            optimizer.step()

        torch.cuda.empty_cache()

The only thing that changes between training and validation is that during training I set validation = False, so I enter in the if condition. And during validation i set the validation variable = True, so I don’t ente rin the if condition.
So, setting validation to False gave me this error :
RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 15.90 GiB total capacity; 14.53 GiB already allocated; 25.75 MiB free; 14.86 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Does someone have an idea to fix it please ?