Your tensor is too small. The caching allocator allocates memory with a minimum size and a block size so it may allocate a bit more memory than the number of elements in your tensors.
Things like cuda ctx, cuda rng state, cudnn ctx, cufft plans, and other gpu memory your other libraries may use are not counted in the torch.cuda.* stats.