Answers.Body: <p>You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.</p>
  
  <p>I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:</p>
  
  <p><strong>Prompt:</strong> </p>
  
  <blockquote>
    <p>I want to generate beautiful music using machine learning. Do you
    think a computer will ever be able to compose something as novel as
    creations of Mozart, Bach, or Chopin?</p>
  </blockquote>
  
  <p><strong>Results</strong> (first four samples, not cherry picked):</p>
  
  <blockquote>
    <p>Yes, computers will probably be able to do it by 2020. I think the key
    for us will be to make music that is interesting and beautiful, so we
    will never really know. My hope would be that computers will become
    far more efficient over time – something that is already happening in
    a lot of ways like in big data – while still being able to do more
    things that they are trained on, like find patterns. I think that's
    where I'd want to use neural networks in the future.</p>
  </blockquote>
  
  <hr>
  
  <blockquote>
    <p>We don't know, but it will probably happen. That's an interesting
    question to ask. Of course we have AI, which can do interesting things
    without human input. It can recognize the world, find objects and
    images, it understands language, we can make games for it. But if you
    imagine an AI being able to write music, will it even be good enough?
    How much better than human players? How much better can someone else
    do? AI will never really be at its best because it will always have to
    come up with creative ways to use and play it. To ask a question like
    it is good enough is to ignore that it is human. You are only allowed
    to ask good questions</p>
  </blockquote>
  
  <hr>
  
  <blockquote>
    <p>The software will help, as it already can do. But it will also allow
    more human creators, which is what I think it should do more of to
    bring music to any genre. In terms of composing, it's a matter of
    creativity in humans in general. But how does it achieve that? Let's
    just say that there are fewer humans composing today than ever before.
    Do you remember the day when, by chance, a computer could write lyrics
    to Mozart's Sonata No. 4, or the day when a human wrote lyrics to
    Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's
    just not true anymore. As we move forward, we are not going to create
    music that human musicians cannot compose. What is your goal as a
    computer music creator?</p>
  </blockquote>
  
  <hr>
  
  <blockquote>
    <p>At the core of this idea is that music is a collection of sounds – it
    was the very first art form, it was the first music that humans
    evolved, a collection of sounds. Music is not an abstract thing of
    sound, but a collection of sounds. You can't write music the way we
    write and compose books and films, because we write and compose with
    sounds in mind. This is where machine learning comes in. You can read
    some books about how to get better music writers or how to improve the
    quality of their music, but you cannot make a difference with this
    kind of information at all. What does music teach you? To write and
    perform beautifully, to convey emotion accurately, and to create
    lasting and rich relationships with</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-19T17:31:30.780Z
Answers.Text: 4 random exemples doesn't constitute a representative metric.
Answers.UserDisplayName: Lucas Morin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-19T18:06:39.330Z
Answers.Text: We don't have any good metrics to evaluate the quality of generated text. If you read papers on the topic you will see that human evaluation remains the best indicator of quality.
Answers.UserDisplayName: MichaelSB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-11T23:14:18.280Z
Answers.LastActivityDate: 2019-06-11T23:14:18.280Z
Answers.UserDisplayName: MichaelSB
Body: <p>You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, <a href="https://blog.openai.com/better-language-models/" rel="nofollow noreferrer">OpenAI</a>, ironically refused to share the whole model fearing dangerous implications. Along the paper, they also published a manifesto to justify their choice: &quot;Better Language Models and Their Implications&quot;. And soon a lot of media were publishing articles discussing the choice and its effectiveness to actually prevent bad implications. I am not here to discuss the ethical components of this choice but the actual performance of the model.</p>
  <p>The model got my attention too and I downloaded the small model to play with. To be honest I am far from impressed by the results. Some times the first paragraph of the produced text appears to make sense, but nine times out of ten it is giberish by the first or the second sentence. Exemples given in the paper seems to be &quot;Lucky&quot; outputs, cherry picked by human hands. Overall, the paper may suffer from a very strong publication bias.</p>
  <p>However, most article we can read on the internet seems to take its powerfulness for granted. <a href="https://www.technologyreview.com/s/612975/ai-natural-language-processing-explained/" rel="nofollow noreferrer">The MIT technology review</a> wrote:</p>
  <blockquote>
  <p>The language model can write like a human</p>
  </blockquote>
  <p><a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction" rel="nofollow noreferrer">The Guardian</a> wrote</p>
  <blockquote>
  <p>When used to simply generate new text, GPT2 is capable of writing plausible passages that match what it is given in both style and subject. It rarely shows any of the quirks that mark out previous AI systems, such as forgetting what it is writing about midway through a paragraph, or mangling the syntax of long sentences.</p>
  </blockquote>
  <p>The model appears generally qualified as a &quot;breakthrough&quot;. These writings do not match my personal experimentation as produced texts are rarely consistent / syntactically correct.</p>
  <p>My question is: without the release of the whole model for ethical reasons, how do we know if the model is really that powerful?</p>
  
Tags: <natural-language-processing><transformer><gpt>
Title: How do we know if GPT-2 is a better language model?
UserDisplayName: Lucas Morin
ThreadText: # How do we know if GPT-2 is a better language model?
  
  (Asked by: username_0 on 2019-02-25)
  
  You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, [OpenAI](https://blog.openai.com/better-language-models/), ironically refused to share the whole model fearing dangerous implications. Along the paper, they also published a manifesto to justify their choice: "Better Language Models and Their Implications". And soon a lot of media were publishing articles discussing the choice and its effectiveness to actually prevent bad implications. I am not here to discuss the ethical components of this choice but the actual performance of the model.
  
  The model got my attention too and I downloaded the small model to play with. To be honest I am far from impressed by the results. Some times the first paragraph of the produced text appears to make sense, but nine times out of ten it is giberish by the first or the second sentence. Exemples given in the paper seems to be "Lucky" outputs, cherry picked by human hands. Overall, the paper may suffer from a very strong publication bias.
  
  However, most article we can read on the internet seems to take its powerfulness for granted. [The MIT technology review](https://www.technologyreview.com/s/612975/ai-natural-language-processing-explained/) wrote:
  
  > The language model can write like a human
  
  [The Guardian](https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction) wrote
  
  > When used to simply generate new text, GPT2 is capable of writing plausible passages that match what it is given in both style and subject. It rarely shows any of the quirks that mark out previous AI systems, such as forgetting what it is writing about midway through a paragraph, or mangling the syntax of long sentences.
  
  The model appears generally qualified as a "breakthrough". These writings do not match my personal experimentation as produced texts are rarely consistent / syntactically correct.
  
  My question is: without the release of the whole model for ethical reasons, how do we know if the model is really that powerful?
  
  **Tags:** natural-language-processing, transformer, gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-11. Score: 1
  
  You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.
  
  I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:
  
  **Prompt:**
  
  > I want to generate beautiful music using machine learning. Do you
  > think a computer will ever be able to compose something as novel as
  > creations of Mozart, Bach, or Chopin?
  
  **Results** (first four samples, not cherry picked):
  
  > Yes, computers will probably be able to do it by 2020. I think the key
  > for us will be to make music that is interesting and beautiful, so we
  > will never really know. My hope would be that computers will become
  > far more efficient over time – something that is already happening in
  > a lot of ways like in big data – while still being able to do more
  > things that they are trained on, like find patterns. I think that's
  > where I'd want to use neural networks in the future.
  
  ---
  
  > We don't know, but it will probably happen. That's an interesting
  > question to ask. Of course we have AI, which can do interesting things
  > without human input. It can recognize the world, find objects and
  > images, it understands language, we can make games for it. But if you
  > imagine an AI being able to write music, will it even be good enough?
  > How much better than human players? How much better can someone else
  > do? AI will never really be at its best because it will always have to
  > come up with creative ways to use and play it. To ask a question like
  > it is good enough is to ignore that it is human. You are only allowed
  > to ask good questions
  
  ---
  
  > The software will help, as it already can do. But it will also allow
  > more human creators, which is what I think it should do more of to
  > bring music to any genre. In terms of composing, it's a matter of
  > creativity in humans in general. But how does it achieve that? Let's
  > just say that there are fewer humans composing today than ever before.
  > Do you remember the day when, by chance, a computer could write lyrics
  > to Mozart's Sonata No. 4, or the day when a human wrote lyrics to
  > Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's
  > just not true anymore. As we move forward, we are not going to create
  > music that human musicians cannot compose. What is your goal as a
  > computer music creator?
  
  ---
  
  > At the core of this idea is that music is a collection of sounds – it
  > was the very first art form, it was the first music that humans
  > evolved, a collection of sounds. Music is not an abstract thing of
  > sound, but a collection of sounds. You can't write music the way we
  > write and compose books and films, because we write and compose with
  > sounds in mind. This is where machine learning comes in. You can read
  > some books about how to get better music writers or how to improve the
  > quality of their music, but you cannot make a difference with this
  > kind of information at all. What does music teach you? To write and
  > perform beautifully, to convey emotion accurately, and to create
  > lasting and rich relationships with
  
  > **Comments:**
  > **username_0**: 4 random exemples doesn't constitute a representative metric.  
  > **username_1**: We don't have any good metrics to evaluate the quality of generated text. If you read papers on the topic you will see that human evaluation remains the best indicator of quality.  
  
  ---
  
  