Hi,
I am trying to figure out why my GPU is crashing. At first I was passing model and the train/val dataloaders to cuda, and got the message" cuda out of memory".
Now I am passing just the model and every batch in the train loop to cuda.
There is something I don’t understand. At the very beginning of my notebook I run this code:
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())

and get:
1024
2097152

I don’t understand where the 2097152 is coming from.
My training loop is as follows:
1- loop on the train loader and calculate the train loss.
2 -loop on the val loader and calculate the val loss.
3 - call a defined score() function on train loader (that loops train loader and predict using the model)
4 - call a defined score() function on val loader.
Everything works fine until step 3. during the score function I receive this error:
CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 2.61 GiB already allocated; 2.10 MiB free; 354.91 MiB cached)

the memory state at the end of step 2 is:
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
3274752
48234496

and Finally here’s what im doing inside the score() function:
    predicted = torch.tensor([])
    Y = torch.tensor([])
    for i, (inputs , targets) in enumerate(train_loader):
        inputs, targets = inputs.cuda(), targets.cuda()
        pred = model(inputs)
        pred = pred.cpu()
        predicted = torch.cat((predicted, pred.float()), 0)
        Y = torch.cat((Y, targets.cpu()), 0)
    y_true = Y.numpy()
    y_pred = predicted.detach().numpy()

and here’s the GPU I have:

Any tips to train my models locally ? I really appreciate your help !