@KnHuq we do occupy some base memory of ~400MB per GPU for cuda context, CUDA RNG context, streams etc. and ~200MB per GPU for CuDNN handles etc.
It depends on GPU model as well. The numbers I quoted are for Volta V100 GPU.