Thanks for the response. According to documentation, each weight, for example W_ir has dimensions (3*hidden_size, hidden_size) assuming unidirectional GRU which is what I have. I set hidden_size=1, so the tensor you see in weight_ih_l0 which has dimensions(3,1)is not W_ir W_iz W_in concatenated. It should refer to just one tensor.