Solved by ptrblck in post #2 
              
              
                The majority of the memory allocation on the GPU are used by the CUDA context, which is created during the first CUDA operation. You could lower is by building from source for your GPU architecture and remove unnecessary libs (such as NCCL in case you don’t need it).