that’s odd, you don’t even use DataParallel in your code sample, and you empty the cache at each iteration…
I’ve seen something in your original code, maybe instead of calling .cuda() on your logits and label tensors, try calling .to('cuda:0'), the same as your model. I have no idea if that would change anything, since it would probably not work if the tensors were on a different GPU than the model, but it’s worth trying.