I’m trying to implement a model for image-text joint embedding, which is trained by optimizing the pariwise ranking loss. However I noticed a problem about LSTM’s last hidden state (often used as the embedding of the input sequence) – its statistical property is a bit weird.
The input batch is (160 sequences, 18 max sequence length) as shown in the picture:

trainbatch-160.png647×554 19.7 KB

and the last hidden state from a fresh (untrained) LSTM looks like this:

trainbatch-rnnfeat-160.png908×741 175 KB

There are obvious vertcial stripes. Word embedding in use is torchtext.vocab.GloVe.
This statistical problem leads to inferior performance and a bad joint embedding space. After some iterations, vertical stipes will also appear in the cnn feature matrix.
Does anyone know how to solve this problem? Thanks in advance.