Might be because the self.word_embedding or q is not in GPU. Since the rest of the code is not available, have you checked if these tensors are in GPU by explicitly setting them to your_tensor.to(torch.device("gpu"))?