LSTM hidden & cell outputs and packed_sequence for variable-length sequence inputs
I have three questions regarding LSTM. Thanks!
supakjk
For a forward RNN, the returned last hidden and cell values are e00 if you don’t use PackedSequence, but they’re ezw if you do. For the backward direction of a bidirectional RNN, they’re axv in both cases, but the RNN will have started at ezw in the PackedSequence case and e00 in the case without it.
jekbradbury
supakjk
cerisara
Yes, the padding is always assumed to be on the right. There are reasons to put the padding on the left if your model is performing computations on the padding tokens and you want to minimize the distortion, but if there are reasons to use left-padding rather than right-padding when the padding tokens won’t be used, we must have overlooked them.
jekbradbury
cerisara
I have a question. What does it mean “padding on the right”? In real situation, “a” is represented by a vector, right? so one sample “x y z 0 0” is represented by a matrix. So what you mean “padding on the right” is “padding several all-zero lines under the useful data”. Am I right?
Nick_Young
Yes. The padding is on the right if the timesteps of the sentence are left to right and you think of the vector of x or a as going into the screen.
jekbradbury
I can not understand how it works. I saw doc in pytorch but I couldn’t figure it out. is there anyone who help me??
JooSung_Yoon
providing initial hidden state is independent of whether you give packed sequences or not. If you dont give initial hidden state, it is initialized to zero.
smth
yukw777