There is some memory used by the cuda driver/manager.
Tensor metadatas are also stored on CPU memory but these should be very small.
Your dataloading uses CPU memory when loading stuff from disk: anything that is loaded from disk has to be in CPU memory before being sent on the GPU.
Do you see some abnormal CPU memory usage?