You could try to update to the latest nightly and recheck it.
Otherwise, I would recommend to use CUDA_VISIBLE_DEVICES to mask all unwanted devices, which will make sure to not use any memory on the masked devices.