MrRobot:


How should I carry the cell state and hidden state along different time step.



Looking at the PyTorch API for inspiration, you can keep it in a local variable and return it at the end.


Whether the models constructed this way could backprop correctly.


Yes.


Is there any standardized dataset I could use to check my implementation is correct.


The usual way is to feed random data through your module and then compare it to a reference implementation (e.g. PyTorch’s or the Numpy one you saw if you prefer).
Best regards
Thomas