I think there was a bug in pytorch 0.3 that was fixed in 0.3.1 where if the dataloader was using pinned memory it created context on a 0-th gpu even if 0-th gpu was not used. Probably that’s what you are seeing. As @ptrblck is suggesting, try updating to pytorch 0.4, this is the latest stable version.