Thanks for the update.
Nevertheless, it seems this line of code is creating the CUDA call:
pos = torch.arange(0, src_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)

Could you set the device to 'cpu' for the sake of debugging here:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

and remove the (proper) if condition?