Ya, In CPU mode you cannot use DataParallel ().
Wrapping a module with DataParallel () simply copies the model over multiple GPUs and puts the results in  device_ids[0] i.e. in 1st of the number of GPUs provided. If you are running in CPU mode, you should simply remove the DataParallel () wrapping. Regarding the documentation line “The code does not need to be changed in CPU-mode.” I guess it means the rest of code( other than model definition) need not be changed. But I am not sure about this and someone else can explain this part.