I’m not sure which input shapes are used and thus cannot completely run the code.
However, the original error is most likely raised in these lines of code:
        z = torch.sigmoid(emb @ self.W_z + h @ self.U_z + self.b_z)
        r = torch.sigmoid(emb @ self.W_r + h @ self.U_r + self.b_r)

since h is initialized as a CPUTensor.
Add the device to init_hidden and try to rerun the code:
    def init_hidden(self, batchSize):
        device = next(self.parameters()).device
        return torch.zeros(1, batchSize, hiddenDimSize, device=device)