I can’t quite reproduce the error since the main training loop hasn’t been posted. Though, with nn.DataParallel, a replica of your model will be created on each device (passed in via device_ids). In your forward function, it looks like you conditionally do things such as .cuda(), .cpu() , which can force tensors to conditionally be on a different device, which can result in this error. If possible, you could try not making your forward function rely on tensors being on specific devices, and instead do this logic in the main training loop.