I figured the problem. The condition in the init_hidden function evaluates to false on always, whether on GPU or CPU and thus, never moves the hidden states parameters to the GPU when run on GPU.
if next(self.parameters()).data.is_cuda:

Any idea why it evaluates to false always?