title: How to use Transformers for text classification?
body: <p>I have two questions about how to use Tensorflow implementation of the Transformers for text classifications. </p>
  
  <ul>
  <li><strong>First</strong>, it seems people mostly used only the encoder layer to do the text classification task. However, encoder layer generates one prediction for each input word. Based on my understanding of transformers, the input to the encoder each time is one word from the input sentence. Then, the attention weights and the output is calculated using the current input word. And we can repeat this process for all of the words in the input sentence. As a result we'll end up with pairs of (attention weights, outputs) for each word in the input sentence. Is that correct? Then how would you use this pairs to perform a text classification?  </li>
  <li><strong>Second</strong>, based on the Tensorflow implementation of transformer <a href="https://www.tensorflow.org/beta/tutorials/text/transformer" rel="noreferrer">here</a>, they embed the whole input sentence to one vector and feed a batch of these vectors to the Transformer. However, I expected the input to be a batch of words instead of sentences based on what I've learned from <a href="http://jalammar.github.io/illustrated-transformer/" rel="noreferrer">The Illustrated Transformer</a></li>
  </ul>
  
  <p>Thank you!</p>