Solved by ptrblck in post #2 
              
              
                This memory is most likely allocated by the CUDA context and not parameters or tensors. 
Depending on the CUDA version, used device etc. you might see an allocation of ~700+MB.