And I found that this only happens when the GRU(or ther RNN layers) was on cpu, and the hidden states contained a Tensor (instead of a cuda Tensor), and I thought that this was a feature specially available for CPU inplementation.