ilyes:

Which is weird! it’s just a one element tensor! Can you please explain why this huge difference between cache and allocated memory.


If I’m not mistaken, cudaMalloc rounds to 2MB blocks in newer GPUs, which is exactly (2097152 / 1024**2), which would explain the minimal cache size.



 ilyes:

One last question: Now as everything is working well, I re-checked my GPU performance and found that utilization is between 3% and 6% Even when I increase the batchsize to 256 the range is 5% to 20% . Why the GPU is not fully used ?


You might have a data loading bottleneck.
Profile your data loading overhead using the code from the ImageNet example 8. If you see some overhead in loading the data, check this post 12 from @rwightman, where he explains some possible workarounds.