Answers.Body: <p>I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the lengths of sentences are higher. So the value of 0.5 doesn't tell much. The fact that you improved accuracy by 0.1 for instance on the same dataset means one method is better than another. Also, people measure perplexity and it's more sensitive to small changes of loss than accuracy. That's why I suggest you to measure perplexity. This metric also depends on lengths of sentences, that's why you should only compare 2 models trained on the same dataset.
  What about GPT and Transformer, they were invented to solve one of the issues of LSTMs (they're not capable to memorize a very large context), that's why transformers would be better for long sentences, but they would also be better for short I think due to their attention mechanist, which has many useful properties.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T15:20:22.927Z
Answers.Text: You mean to suggest perplexity, I think you made an error when writing "That's why I suggest you to measure accuracy."
Answers.UserDisplayName: SajanGohil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T15:47:13.183Z
Answers.Text: Sorry, yeah, I meant perplexity
Answers.UserDisplayName: Michał Słodki
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T15:07:07.023Z
Answers.LastActivityDate: 2020-09-13T15:47:54.627Z
Answers.LastEditDate: 2020-09-13T15:47:54.627Z
Answers.UserDisplayName: Michał Słodki
Body: <p>I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 characters). So far, I have tried LSTM's with 1-3 layers and inputs as subwords and characters. The results so far are not that great, I am getting ~0.5 crossentropy loss and ~50% accuracy.</p>
  <p>My inputs are like a sliding window with prepadding, (eg. (for a batch) <code>Inputs = [[0,0,0,1], [0,0,1,2]...[n-4,..n-1]]</code>, <code>outputs=[[0,0,1,2], ...[n-3,n-2,n-1,n]]</code>) where 0 is padding, <code>1</code> is the start token and <code>n</code> is the end token. Outputs are 1 hot encoded.</p>
  <p>The model is an embedding layer, few lstm and dropout layers, followed by time distributed dense and then a dense layer.</p>
  <p>My doubt is, is accuracy a right metric, I am using it because at the end, I am making a classification for 4 output values. Another one is, will a transformer be suitable for this, since I want to generate small sentences, (which are nouns) and models like GPT/ Bert are more suitable for capturing dependency between long sentences.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-13T15:25:57.330Z
Comments.Text: Hi. Can please put your main question in the title?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-13T15:28:12.390Z
Comments.Text: @nbro updated, is that okay?
Comments.UserDisplayName: SajanGohil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-13T15:28:55.370Z
Comments.Text: That seems more explicit and specific. Thanks!
Comments.UserDisplayName: nbro
Tags: <natural-language-processing><long-short-term-memory><attention><language-model>
Title: Appropriate metric and approach for natural language generation for small sentences
UserDisplayName: SajanGohil
ThreadText: # Appropriate metric and approach for natural language generation for small sentences
  
  (Asked by: username_0 on 2020-09-13)
  
  I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 characters). So far, I have tried LSTM's with 1-3 layers and inputs as subwords and characters. The results so far are not that great, I am getting ~0.5 crossentropy loss and ~50% accuracy.
  
  My inputs are like a sliding window with prepadding, (eg. (for a batch) `Inputs = [[0,0,0,1], [0,0,1,2]...[n-4,..n-1]]`, `outputs=[[0,0,1,2], ...[n-3,n-2,n-1,n]]`) where 0 is padding, `1` is the start token and `n` is the end token. Outputs are 1 hot encoded.
  
  The model is an embedding layer, few lstm and dropout layers, followed by time distributed dense and then a dense layer.
  
  My doubt is, is accuracy a right metric, I am using it because at the end, I am making a classification for 4 output values. Another one is, will a transformer be suitable for this, since I want to generate small sentences, (which are nouns) and models like GPT/ Bert are more suitable for capturing dependency between long sentences.
  
  **Tags:** natural-language-processing, long-short-term-memory, attention, language-model
  
  > **Comments:**
  > **username_1**: Hi. Can please put your main question in the title?  
  > **username_0**: @username_1 updated, is that okay?  
  > **username_1**: That seems more explicit and specific. Thanks!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-09-13. Score: 1
  
  I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the lengths of sentences are higher. So the value of 0.5 doesn't tell much. The fact that you improved accuracy by 0.1 for instance on the same dataset means one method is better than another. Also, people measure perplexity and it's more sensitive to small changes of loss than accuracy. That's why I suggest you to measure perplexity. This metric also depends on lengths of sentences, that's why you should only compare 2 models trained on the same dataset.
  What about GPT and Transformer, they were invented to solve one of the issues of LSTMs (they're not capable to memorize a very large context), that's why transformers would be better for long sentences, but they would also be better for short I think due to their attention mechanist, which has many useful properties.
  
  > **Comments:**
  > **username_0**: You mean to suggest perplexity, I think you made an error when writing "That's why I suggest you to measure accuracy."  
  > **username_2**: Sorry, yeah, I meant perplexity  
  
  ---
  
  