Thanks again for taking a look into this.
The original problem has a vocabulary sized 50000 and hidden size 1000, I think it should be enough model complexity to overfit the data? I apologize that I’m focusing too much in the -1/1 problem of the hidden units in this toy example and may not realize the model complexity. But, did you observe that the hidden unit saturated? I wrote a quick tf version 3 for this toy example just now . Though the model complexity seems not enough too with the same setting of the original post, but the hidden states just, never goes to -1/1.