It is reserved, not allocated. The reserved memory can increase due to many reasons (more code in the binary, caching allocator behavior, different GPU, different cudnn, etc). The allocated one corresponds to only Tensors (and this one should be constant).