Solved by tom in post #2 
              
              
                Tensors come with metadata (information about stride, size, type, …) in CPU memory and you probably kept a lot of these around, possibly also some graphs to be used in gradient calculations. When you only have a scalar value, the CPU memory usage probably exceeds the GPU memory use by quite a margin…