The first CUDA operation will create the CUDA context, which will use ~700-800MB of device memory.
Are you seeing a difference in the memory usage while running the training or only during the model loading?