Yes, it’s because it’s not large enough.
Try this one:
torch.rand(20000,20000).cuda()

It will allocate nearly 2GB.
P.S. print(“Allocated:”, round(torch.cuda.memory_allocated(0)/10243,1), “GB”) this clause is not correct and I just copied it from your code to say that it’s not zero.