Hi @ptrblck,  I am currently having the GPU memory leakage problem (during evaluation) that
(1)   the GPU memory usage increased during evaluation, and
(2)   it is not fully cleared after all variables have been deleted, and i have also cleared the memory using torch.cuda.empty_cache() and gc.collect().
Do you have any idea on why the GPU remains occupied after the evaluation stage? and this problem occurs even if I set the batch size to 1.
My evaluation function is as follows:
 with torch.no_grad():
        
        for batch_idx, (x, y) in enumerate(loader_gallery):

            x = x.to(device)

            x, xx = model(x)
            xx = submodel(xx)

            gallery_fea_1 = torch.cat((gallery_fea_1, x.cpu()), 0)
            gallery_fea_2 = torch.cat((gallery_fea_2, xx.cpu()), 0)
            gallery_label = torch.cat((gallery_label, y.cpu()), 0)

            del x, xx, y
            time.sleep(0.0001)


gpu1081×303 8.15 KB

Thank you.
Sincerely.