problem solved
I did it wrong when returning the hidden_seq
hidden_seq = torch.cat(tuple(hidden_seq), 0).reshape(batch_size, seq_size, hidden_size)

this line is wrong, cuz I orgnized it wrong
The right one should be
hidden_seq = torch.cat(hidden_seq,dim=2).reshape(batch_size,seq_size,hidden_size)