Did you use DataParallel or how do you use more than 1 GPU?
If so, did you keep batch_size=1 for multiple GPUs?
DataParallel tries to split the batch between all GPUs, so batch_size=1 could be problematic.
However, the error message seems to point to another issue. Could you verify my assumptions?