Hi @hzzzm,
do you pad your input sequences? If yes, I guess that comes from LSTM’s nature, even tho this might occur more obviously when working with GRUs.863×671
When the input becomes 0 (that’s what padding does with value 0), the added up previous hidden and current input is smaller than ‘normal’ (with ‘normal’ inputs). Therefore, the first sigmoid’s output will be lower and therefore the cell state will forget more. However, that might not even be the biggest issue. By looking at the input gate, we can see, that the zero - centred tanh multiplied with the sigmoid’s output adds less to the cell state than normally. The same goes for the output gate, as the output gate’s sigmoid function outputs lower values than ‘normally’. Therefore, even the hidden state gets closer and closer to zero, as well as the cell state with it. Furthermore, all these operations are accompanied by weights, which are supporting this convergence.
Anybody, please feel free to correct me.
Regards,
Unity05