Answers.Body: <p>So after doing a bit of research, I finally found out why the model is not working at all when I change the LSTM to Bi-LSTM.</p>
  <p>The task of the learning is Next Word Prediction for each cell of LSTM. When you have a Uni-directional LSTM, this is inherently a tough task for the model to learn good representations that can help it generate the next word with enough confident.</p>
  <p>What happens when you change the model to a Bi-LSTM is that, if you concatenate the forward and backward values of each cell together, you have now the information of the very next word you wanted to predict via the backward route.</p>
  <p>To alleviate this issue, <a href="https://arxiv.org/abs/1604.00790" rel="nofollow noreferrer">Wang et al.</a> propose to do prediction on forward and backward routes separately while training the data, and for generating, see which route has more confidence in its generated caption.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-10T17:26:51.427Z
Answers.LastActivityDate: 2021-06-10T17:26:51.427Z
Answers.UserDisplayName: Farhood ET
Body: <p>Currently I'm dealing with an assignment that made us implement the network mentioned in <a href="https://arxiv.org/abs/1502.03044" rel="nofollow noreferrer">this paper</a>. The network has an architecture similar to this:</p>
  <p><a href="https://i.sstatic.net/rWWnE.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rWWnE.jpg" alt="Network Architecture" /></a></p>
  <p>As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:</p>
  <pre><code>def predict(self, image, max_len = 30):
        output = []
        hidden = None
        inputs = self.encoder(image).unsqueeze(1) # Image features
        for i in range(max_len): # Recursively feed generated words to LSTM 
          lstm_out, hidden = self.decoder.lstm(inputs,hidden)
          output_vocab = self.decoder.fc(lstm_out)    
          output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy()
          words_indices = output_vocab.argsort(axis=1).squeeze()
          word = words_indices[-1]
          if word == self.unk_token_index:
              word = indices[-2]
          output.append(int(word))
          if word == self.end_token_index:
            break  
          inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device))
        return output
  </code></pre>
  <p>The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.</p>
  <p>This is a sample caption generated by Bi-LSTM:</p>
  <p><a href="https://i.sstatic.net/rzLMX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rzLMX.png" alt="Sample generated by ‌BiLSTM, picture from Flickr8k dataset." /></a></p>
  <p>This is a sample caption generated by UniLSTM:</p>
  <p><a href="https://i.sstatic.net/9KRLN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9KRLN.png" alt="Sample generated by ‌UniLSTM, picture from Flickr8k dataset." /></a></p>
  <p>My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.</p>
  
Tags: <deep-learning><natural-language-processing><computer-vision><long-short-term-memory><pytorch>
Title: Changing a CNN-LSTM image captioning architecture to use BiLSTMs
UserDisplayName: Farhood ET
ThreadText: # Changing a CNN-LSTM image captioning architecture to use BiLSTMs
  
  (Asked by: username_0 on 2021-06-07)
  
  Currently I'm dealing with an assignment that made us implement the network mentioned in this paper. The network has an architecture similar to this:
  
  As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:
  def predict(self, image, max_len = 30):
        output = []
        hidden = None
        inputs = self.encoder(image).unsqueeze(1) # Image features
        for i in range(max_len): # Recursively feed generated words to LSTM 
          lstm_out, hidden = self.decoder.lstm(inputs,hidden)
          output_vocab = self.decoder.fc(lstm_out)    
          output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy()
          words_indices = output_vocab.argsort(axis=1).squeeze()
          word = words_indices[-1]
          if word == self.unk_token_index:
              word = indices[-2]
          output.append(int(word))
          if word == self.end_token_index:
            break  
          inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device))
        return output
  
  The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.
  This is a sample caption generated by Bi-LSTM:
  
  This is a sample caption generated by UniLSTM:
  
  My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.
  
  **Tags:** deep-learning, natural-language-processing, computer-vision, long-short-term-memory, pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-06-10. Score: 0 | Accepted answer
  
  So after doing a bit of research, I finally found out why the model is not working at all when I change the LSTM to Bi-LSTM.
  
  The task of the learning is Next Word Prediction for each cell of LSTM. When you have a Uni-directional LSTM, this is inherently a tough task for the model to learn good representations that can help it generate the next word with enough confident.
  
  What happens when you change the model to a Bi-LSTM is that, if you concatenate the forward and backward values of each cell together, you have now the information of the very next word you wanted to predict via the backward route.
  
  To alleviate this issue, [Wang et al.](https://arxiv.org/abs/1604.00790) propose to do prediction on forward and backward routes separately while training the data, and for generating, see which route has more confidence in its generated caption.
  
  
  ---
  
  