Answers.Body: <p>this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation:</p>
  <p><a href="http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/" rel="noreferrer">http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/</a></p>
  <blockquote>
  <pre><code>although he had already eaten a large meal, he was still very hungry
  </code></pre>
  <p>As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in good shape for generation.</p>
  <p>This failed. BERT predicted “much” as the last word. Maybe this is because BERT thinks the absence of a period means the sentence should continue. Maybe it’s just so used to complete sentences it gets confused. I’m not sure.</p>
  <p>One might argue that we should continue predicting after “much”. Maybe it’s going to produce something meaningful. To that I would say: first, this was meant to be a dead giveaway, and any human would predict “hungry”. Second, I tried it, and it keeps predicting dumb stuff. After “much”, the next token is “,”.</p>
  <p>So, at least using these trivial methods, BERT can’t generate text.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-24T21:15:53.543Z
Answers.LastActivityDate: 2019-03-24T21:15:53.543Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: stuart
Answers.Body: <p>No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). <a href="https://github.com/google-research/bert/issues/35" rel="nofollow noreferrer">Because of bi-directionality of BERT, BERT cannot be used as a language model.</a> If it cannot be used as language model, I don't see how you can generate a sentence using BERT.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-31T07:41:24.390Z
Answers.Text: Do alternate options exist for this?
Answers.UserDisplayName: Shamit Bagchi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T23:51:19.860Z
Answers.Text: My answer is no longer correct. You may want to accept @soloice 's answer
Answers.UserDisplayName: Astariul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-22T21:53:10.237Z
Answers.Text: Could one generate a sentence that doesn't make sense?  My understanding is the possible combinations of natural language words in a string is infinite (definitely countably infinite, but possibly uncountably infinite.)  Although most of the text generated would lack semantic significance, that random generation would also produce plenty of meaningful natural language sentences, given sufficient time and space?
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-07T13:27:56.953Z
Answers.Text: @Astariul Your answer was never correct, as it was possible all along to generate text with BERT; we just hadn’t realized it yet ;)
Answers.UserDisplayName: HelloGoodbye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-26T01:01:44.943Z
Answers.LastActivityDate: 2019-02-17T11:23:00.753Z
Answers.LastEditDate: 2019-02-17T11:23:00.753Z
Answers.UserDisplayName: Astariul
Answers.Body: <p>For newbies, NO.</p>
  
  <p>Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.</p>
  
  <hr>
  
  <p>For advanced researchers, YES.</p>
  
  <p>You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition). Though the text generation quality is hard to control.</p>
  
  <p>Here's the technical report <a href="https://arxiv.org/abs/1902.04094" rel="noreferrer">BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model</a>, its <a href="https://sites.google.com/site/deepernn/home/blog/amistakeinwangchoberthasamouthanditmustspeakbertasamarkovrandomfieldlanguagemodel" rel="noreferrer">errata</a> and the <a href="https://github.com/nyu-dl/bert-gen" rel="noreferrer">source code</a>.</p>
  
  <hr>
  
  <p>In summary:</p>
  
  <ul>
  <li>If you would like to do some research in the area of decoding with
  BERT, there is a huge space to explore</li>
  <li>If you would like to generate
  high quality texts, personally I recommend you to check <a href="https://blog.openai.com/better-language-models/" rel="noreferrer">GPT-2</a>.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-07T13:20:13.017Z
Answers.Text: Wouldn’t it be easier for the model, though, to generate high quality text if you let it generate the words from left to right, just like with a language model, instead of having it generate the words in an arbitrary order? I can just imagine if I had to write all words in an arbitrary order whenever I wrote a sentence; it would be pretty difficult to know which word I should write at position seven, for example, if I hadn’t written any word before that.
Answers.UserDisplayName: HelloGoodbye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T05:31:04.100Z
Answers.LastActivityDate: 2019-10-13T09:55:34.357Z
Answers.LastEditDate: 2019-10-13T09:55:34.357Z
Answers.UserDisplayName: soloice
Body: <p>I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the <a href="https://github.com/karpathy/char-rnn" rel="noreferrer">CharRNN</a>, which uses RNN to generate a sequence of words.</p>
  
  <p>However, <a href="https://arxiv.org/abs/1810.04805" rel="noreferrer">BERT</a> has come out several weeks ago and is very powerful. Therefore, I am wondering whether this task can also be done with the help of BERT? I am a new learner in this field, and thank you for any advice!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-20T16:17:31.353Z
Comments.Text: Can OpenAI GPT be used for this? I believe OpenAI GPT has different architecture and is used for text generation
Comments.UserDisplayName: Sandeep Bhutani
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-23T00:27:32.203Z
Comments.Text: I believe CharRNN is definitely not SOTA, due to the limited context length, from working at the scale of characters. Instead, there is work on subwords, and byte-pair encodings
Comments.UserDisplayName: information_interchange
Tags: <neural-networks><deep-learning><natural-language-processing><bert><text-generation>
Title: Can BERT be used for sentence generating tasks?
UserDisplayName: ch271828n
ThreadText: # Can BERT be used for sentence generating tasks?
  
  (Asked by: username_0 on 2018-11-24)
  
  I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the [CharRNN](https://github.com/karpathy/char-rnn), which uses RNN to generate a sequence of words.
  
  However, [BERT](https://arxiv.org/abs/1810.04805) has come out several weeks ago and is very powerful. Therefore, I am wondering whether this task can also be done with the help of BERT? I am a new learner in this field, and thank you for any advice!
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, bert, text-generation
  
  > **Comments:**
  > **username_1**: Can OpenAI GPT be used for this? I believe OpenAI GPT has different architecture and is used for text generation  
  > **username_2**: I believe CharRNN is definitely not SOTA, due to the limited context length, from working at the scale of characters. Instead, there is work on subwords, and byte-pair encodings  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_8 on 2019-02-17. Score: 36 | Accepted answer
  
  For newbies, NO.
  
  Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.
  
  ---
  
  For advanced researchers, YES.
  
  You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition). Though the text generation quality is hard to control.
  
  Here's the technical report [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://arxiv.org/abs/1902.04094), its [errata](https://sites.google.com/site/deepernn/home/blog/amistakeinwangchoberthasamouthanditmustspeakbertasamarkovrandomfieldlanguagemodel) and the [source code](https://github.com/nyu-dl/bert-gen).
  
  ---
  
  In summary:
  
  * If you would like to do some research in the area of decoding with
    BERT, there is a huge space to explore
  * If you would like to generate
    high quality texts, personally I recommend you to check [GPT-2](https://blog.openai.com/better-language-models/).
  
  > **Comments:**
  > **username_7**: Wouldn’t it be easier for the model, though, to generate high quality text if you let it generate the words from left to right, just like with a language model, instead of having it generate the words in an arbitrary order? I can just imagine if I had to write all words in an arbitrary order whenever I wrote a sentence; it would be pretty difficult to know which word I should write at position seven, for example, if I hadn’t written any word before that.  
  
  ---
  
  ### Answer by username_3 on 2019-03-24. Score: 10
  
  this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation:
  
  <http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/>
  
  > ```
  > although he had already eaten a large meal, he was still very hungry
  >
  > ```
  >
  > As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in good shape for generation.
  >
  > This failed. BERT predicted “much” as the last word. Maybe this is because BERT thinks the absence of a period means the sentence should continue. Maybe it’s just so used to complete sentences it gets confused. I’m not sure.
  >
  > One might argue that we should continue predicting after “much”. Maybe it’s going to produce something meaningful. To that I would say: first, this was meant to be a dead giveaway, and any human would predict “hungry”. Second, I tried it, and it keeps predicting dumb stuff. After “much”, the next token is “,”.
  >
  > So, at least using these trivial methods, BERT can’t generate text.
  
  
  ---
  
  ### Answer by username_4 on 2018-11-26. Score: 2
  
  No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). [Because of bi-directionality of BERT, BERT cannot be used as a language model.](https://github.com/google-research/bert/issues/35) If it cannot be used as language model, I don't see how you can generate a sentence using BERT.
  
  > **Comments:**
  > **username_4**: My answer is no longer correct. You may want to accept @username_8 's answer  
  > **username_5**: Do alternate options exist for this?  
  > **username_6**: Could one generate a sentence that doesn't make sense?  My understanding is the possible combinations of natural language words in a string is infinite (definitely countably infinite, but possibly uncountably infinite.)  Although most of the text generated would lack semantic significance, that random generation would also produce plenty of meaningful natural language sentences, given sufficient time and space?  
  > **username_7**: @username_4 Your answer was never correct, as it was possible all along to generate text with BERT; we just hadn’t realized it yet ;)  
  
  ---
  
  