You could send your gates to the fused pointwise backend and recalculate hy. Would give some gains… see https://github.com/pytorch/pytorch/blob/ceb4f84d12304d03a6a46693e54390869c0c208e/torch/nn/_functions/rnn.py#L23-L28 208
To get really strong performance one would want to merge mean and std into a single kernel, and then

x = (input - mean) / (std + self.eps)
x * self.gain.expand_as(x) + self.bias.expand_as(x)

into a single kernel. Which is a sizable effort.