CUDA context are created on both GPU 0 and GPU 1. All further operations, if only involving GPU 1 tensors, will not use memory on GPU 0.
The context thing is fixed on GitHub master. For a workaround you can set CUDA_VISIBLE_DEVICES.