title: How to do sequence classification with pytorch nn.Transformer?
body: <p>I am doing a sequence classification task using <code>nn.TransformerEncoder()</code>. Whose pipeline is similar to <code>nn.LSTM()</code>.</p>
  
  <p>I have tried several temporal features fusion methods:</p>
  
  <ol>
  <li><p>Selecting the final outputs as the representation of the whole sequence.</p></li>
  <li><p>Using an affine transformation to fuse these features.</p></li>
  <li><p>Classifying the sequence frame by frame, and then select the max values to be the category of the whole sequence.</p></li>
  </ol>
  
  <p>But, all these 3 methods got a terrible accuracy, only <strong>25%</strong> for 4 categories classification. While using nn.LSTM with the last hidden state, I can achieve <strong>83%</strong> accuracy easily. I tried plenty of hyperparameters of <code>nn.TransformerEncoder()</code>, but without any improvement for the accuracy.</p>
  
  <p>I have no idea about how to adjust this model now. Could you give me some practical advice? Thanks.</p>
  
  <p>For <code>LSTM</code>: the <code>forward()</code> is:</p>
  
  <pre class="lang-py prettyprint-override"><code>    def forward(self, x_in, x_lengths, apply_softmax=False):
  
          # Embed
          x_in = self.embeddings(x_in)
  
          # Feed into RNN
          out, h_n = self.LSTM(x_in) #shape of out: T*N*D
  
          # Gather the last relevant hidden state
          out = out[-1,:,:] # N*D
  
          # FC layers
          z = self.dropout(out)
          z = self.fc1(z)
          z = self.dropout(z)
          y_pred = self.fc2(z)
  
          if apply_softmax:
              y_pred = F.softmax(y_pred, dim=1)
          return y_pred
  </code></pre>
  
  <p>For <code>transformer</code>:</p>
  
  <pre class="lang-py prettyprint-override"><code>    def forward(self, x_in, x_lengths, apply_softmax=False):
  
          # Embed
          x_in = self.embeddings(x_in)
  
          # Feed into RNN
          out = self.transformer(x_in)#shape of out T*N*D
  
          # Gather the last relevant hidden state
          out = out[-1,:,:] # N*D
  
          # FC layers
          z = self.dropout(out)
          z = self.fc1(z)
          z = self.dropout(z)
          y_pred = self.fc2(z)
  
          if apply_softmax:
              y_pred = F.softmax(y_pred, dim=1)
          return y_pred
  </code></pre>