With DataParallel, you accumulate the gradients from all GPUs on one of the main GPUs temporarily. Sometimes it can be an issue if all your GPUs use up a lot of memory (then this will basically result in an out-of-memory error). So, currently it is not possible to use a GPU for accumulating the gradients that is not also already used during training and is already holding a model in memory. In practice, I haven’t encountered an issue with that yet but it could be nicer (although maybe a bit slower) to use a dedicated GPU for the gradient accumulation such that memory spikes can be avoided