Sorry for the late reply! I just realized that
1024
2097152

is coming from these two lines:
pos_weights = class_weight["1"] * torch.ones(1, dtype = torch.float32, device = 'cuda')
neg_weights = class_weight["0"] * torch.ones(1, dtype = torch.float32, device = 'cuda')

Which is weird! it’s just a one element tensor! Can you please explain why this huge difference between cache and allocated memory.
You are right! Not doing detach() on pred was causing the problem. Well apparently I missed the fact that the graphs were saved for backprop on every iteration. Now everything works well.
In score() function I am calculating y_pred, and then return some scores to my training loop ( accuracy, roc-auc, f1 …)
One last question: Now as everything is working well, I re-checked my GPU performance and found that utilization is between 3% and 6% Even when I increase the batchsize to 256 the range is 5% to 20%. Why the GPU is not fully used ?