Hm, in that case, instead of wasting one GPU, it is maybe cheaper to just compute the loss on the CPU. I.e. you could try
    features = features.to(device)
    targets = targets
    logits = model(features)
    cost = F.cross_entropy(logits.to(torch.device('cpu'), targets)

where “model” is still a model via dataparallel.
Generally though, I would recommend scaling the batchsize by 1/num_cuda_devices if you use dataparallel and want results that are somewhat comparable to the 1-GPU version.