We call .float() and .long() quite a few times. That explains it. I didn’t that that it calls .to internally (see https://pytorch.org/docs/stable/tensors.html#torch.Tensor.long 5).
Do you happen to know why these calls bother the cpu so much? The tensors they are called upon all reside on a cuda device.