Hey,
Side note, you might want to use the builtin KL: torch.nn.functional — PyTorch 1.8.0 documentation 2
From your code, I think the main issue is that you’re using numpy arrays (which cannot be on GPU).
You might want to keep everything as Tensors and when you create new Tensors, you can pass the device= kwarg. In this case, you want to match the device of the inputs I guess so attn_val.device.