Solved by ptrblck in post #2 
              
              
                The majority of the mentioned used memory comes from the creation of the CUDA context, which cannot be freed. You should also be able to see this memory usage by creating an empty CUDATensor. 
The caching allocator will reserve some memory which will be used later and you cannot disable it at the …