Answers.Body: <p>This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more data than RNNs to reach good text quality.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T11:58:18.187Z
Answers.LastActivityDate: 2021-09-20T11:58:18.187Z
Answers.UserDisplayName: Langueneers
Body: <p>I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder</p>
  <pre><code>class TransformerLM_1(nn.Module):
  
      def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', 
                   pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        
          super(TransformerLM_1, self).__init__()
        
          self.head = head
          self.embedding_size = embedding_size
          self.vocab_size = vocab_size
          self.device = device
          self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
          self.postional_encoding = PostionalEncoding(embedding_size, device)
          self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
          self.out_linear = nn.Linear(self.embedding_size, vocab_size)
          self.dropout = dropout
          self.pad_idx = pad_idx
          self.start_idx = start_idx
          self.end_idx = end_idx
          self.unk_idx = unk_idx
          self.device = device
  
      
      def make_src_mask(self, src_sz):
          mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
          mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
          mask = mask.to(self.device)
          return mask
  
      def forward(self, x):
          dec_in = x.clone()[:, :-1]
          src_mask = self.make_src_mask(dec_in.size()[1])
          src = self.embed(dec_in)
          src = self.postional_encoding(src) 
          src = src.transpose(0,1)
          transformer_out = self.decoder(src, src_mask)
          out = self.out_linear(transformer_out)
          return out
  </code></pre>
  <p>I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.</p>
  <p>Here is sample generated text with the expected</p>
  <pre><code>Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some <span class="math-container">$ n million &lt;eos&gt; 
  Predicted: some &lt;unk&gt; been the been &lt;unk&gt; not be $</span> the total has was the may has &lt;unk&gt; the that that be to the &lt;unk&gt; the 
  
  Expected: citicorp and chase are attempting to put together a new lower bid &lt;eos&gt; 
  Predicted: a are &lt;unk&gt; carries n't to the together with &lt;unk&gt; jersey than 
  
  Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says &lt;eos&gt; 
  Predicted: &lt;unk&gt; ' s &lt;unk&gt; comeback money of the in mr to their &lt;unk&gt; and of &lt;unk&gt; &lt;unk&gt; or or &lt;unk&gt; the money 
  
  Expected: moreover while asian and middle eastern investors &lt;unk&gt; gold and help &lt;unk&gt; its price silver does n't have the same &lt;unk&gt; dealers say &lt;eos&gt; 
  Predicted: the production the routes &lt;unk&gt; of its 
  
  Expected: a board of control spokesman said the board had not seen the claim and declined to comment &lt;eos&gt; 
  Predicted: the board said declined of said 
  
  Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected &lt;eos&gt; 
  Predicted: the claims markets said its was n &lt;unk&gt; to sell insolvent of was n't disclosed to sell its plan 
  
  Expected: similarly honda motor co . ' s sales are so brisk that workers &lt;unk&gt; they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity &lt;eos&gt; 
  Predicted: the honda ' credit . s s &lt;unk&gt; 
  
  Expected: we expect a big market in the future so in the long term it will be profitable &lt;eos&gt; 
  Predicted: it can it &lt;unk&gt; board 
  
  Expected: u . k . composite or &lt;unk&gt; insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading &lt;eos&gt; 
  Predicted: the . s . s trading sell said which &lt;unk&gt; traders market said the be able in the the earthquake 
  
  Expected: this will require us to define and &lt;unk&gt; what is necessary or appropriate care &lt;eos&gt; 
  Predicted: &lt;unk&gt; is be the $ &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; &lt;unk&gt; is the to &lt;unk&gt; and or 
  </code></pre>
  <p>As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?</p>
  <p>EDIT</p>
  <p>This is one example that caught my eye</p>
  <pre><code>Expected: also the big board met with angry stock specialists &lt;eos&gt; 
  Predicted: also met specialists board met the stock big with after 
  </code></pre>
  <p>Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-29T11:18:54.677Z
Comments.Text: I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ?  If so, then do you think, it can be used for sentence generation ?
Comments.UserDisplayName: Murugesh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-29T13:41:57.320Z
Comments.Text: No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. https://pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense.
Comments.UserDisplayName: Hari Krishnan
Tags: <natural-language-processing><pytorch><transformer><text-generation>
Title: Transformer Language Model generating meaningless text
UserDisplayName: Hari Krishnan
ThreadText: # Transformer Language Model generating meaningless text
  
  (Asked by: username_0 on 2020-09-28)
  
  I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder
  
  ```
  class TransformerLM_1(nn.Module):
  
      def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', 
                   pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        
          super(TransformerLM_1, self).__init__()
        
          self.head = head
          self.embedding_size = embedding_size
          self.vocab_size = vocab_size
          self.device = device
          self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
          self.postional_encoding = PostionalEncoding(embedding_size, device)
          self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
          self.out_linear = nn.Linear(self.embedding_size, vocab_size)
          self.dropout = dropout
          self.pad_idx = pad_idx
          self.start_idx = start_idx
          self.end_idx = end_idx
          self.unk_idx = unk_idx
          self.device = device
  
      
      def make_src_mask(self, src_sz):
          mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
          mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
          mask = mask.to(self.device)
          return mask
  
      def forward(self, x):
          dec_in = x.clone()[:, :-1]
          src_mask = self.make_src_mask(dec_in.size()[1])
          src = self.embed(dec_in)
          src = self.postional_encoding(src) 
          src = src.transpose(0,1)
          transformer_out = self.decoder(src, src_mask)
          out = self.out_linear(transformer_out)
          return out
  
  ```
  
  I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.
  
  Here is sample generated text with the expected
  
  ```
  Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos> 
  Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the 
  
  Expected: citicorp and chase are attempting to put together a new lower bid <eos> 
  Predicted: a are <unk> carries n't to the together with <unk> jersey than 
  
  Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos> 
  Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money 
  
  Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos> 
  Predicted: the production the routes <unk> of its 
  
  Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos> 
  Predicted: the board said declined of said 
  
  Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos> 
  Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan 
  
  Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos> 
  Predicted: the honda ' credit . s s <unk> 
  
  Expected: we expect a big market in the future so in the long term it will be profitable <eos> 
  Predicted: it can it <unk> board 
  
  Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos> 
  Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake 
  
  Expected: this will require us to define and <unk> what is necessary or appropriate care <eos> 
  Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or 
  
  ```
  
  As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?
  
  EDIT
  
  This is one example that caught my eye
  
  ```
  Expected: also the big board met with angry stock specialists <eos> 
  Predicted: also met specialists board met the stock big with after 
  
  ```
  
  Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.
  
  **Tags:** natural-language-processing, pytorch, transformer, text-generation
  
  > **Comments:**
  > **username_1**: I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ?  If so, then do you think, it can be used for sentence generation ?  
  > **username_0**: No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. https://pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-09-20. Score: 1
  
  This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more data than RNNs to reach good text quality.
  
  
  ---
  
  