Answers.Body: <p>You could handwrite different templates and choose probabilistically, according to writing style or pragmatic effects like irony and so on, but that very much depends on the domain.
  If you have tabular data, from which you want to generate text, you should probably forget about GPT and so on. You only have few control (despite copy mechanism) over the generation process, you actually predict the next most probable word sequence for a given length - GPTs don't author coherent text across paragraphs, especially not when text length is more than a few hundred words.</p>
  <p>Check the linguistic counterpart (<a href="https://arxiv.org/abs/1703.09902" rel="nofollow noreferrer">https://arxiv.org/abs/1703.09902</a>) to end2end generation systems. Breaking up the networks, pipelining again and using networks for controllable tasks.
  e.g. build a module that selects which attributes to produce. Create RDF triples from the column head words and values in your data base. Take a Text to text Transfer model (Google's T5) and transform into surface text. You should also have a look into the webNLG challenge (<a href="https://webnlg-challenge.loria.fr/challenge_2020/" rel="nofollow noreferrer">https://webnlg-challenge.loria.fr/challenge_2020/</a>). This might help. Much of this is still open research. I am quite busy in this topic, so feel free to ask.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-22T19:36:33.087Z
Answers.Text: Thx for your reply! I am also a researcher. Is there a way to get in touch with you?
Answers.UserDisplayName: Carol.Kar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-23T12:14:57.273Z
Answers.Text: VIa Mail? Don't know whether stackexchange offers private messages, I think these comments will soon get deleted:)
Answers.UserDisplayName: Langueneers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-25T12:49:39.360Z
Answers.Text: Btw if you are interested for an exchange message me on maggregor.samsa@gmail.com
Answers.UserDisplayName: Carol.Kar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T10:56:24.700Z
Answers.LastActivityDate: 2021-09-20T10:56:24.700Z
Answers.UserDisplayName: Langueneers
Answers.Body: <p>The state of the art in text generation is the GPT model. <a href="https://en.wikipedia.org/wiki/GPT-3" rel="nofollow noreferrer">GPT-3</a>, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. <a href="https://www.technologyreview.com/2020/07/20/1005454/openai-machine-learning-language-generator-gpt-3-nlp/" rel="nofollow noreferrer">This</a> article and <a href="https://www.technologyreview.com/2020/10/08/1009845/a-gpt-3-bot-posted-comments-on-reddit-for-a-week-and-no-one-noticed/" rel="nofollow noreferrer">this</a> one should give you an example of how powerful it is at text generation.</p>
  <p>GPT is a <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="nofollow noreferrer">transformer</a> based architecture, somewhat like <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT</a>. The main difference is it only takes into account left context, which is why it is so well suited for text generation.</p>
  <p>GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.</p>
  <p>Here is a <a href="https://minimaxir.com/2019/09/howto-gpt2/" rel="nofollow noreferrer">link</a> to a tutorial explaining the basics to get you started with GPT-2.</p>
  <p>If you are interested in diving into the relevant research papers:</p>
  <p>Here are the open AI papers on GPT 1-3</p>
  <ol>
  <li><a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf" rel="nofollow noreferrer">Improving Language Understanding by Generative Pre-Training</a></li>
  <li><a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" rel="nofollow noreferrer">Language Models are Unsupervised Multitask Learners</a></li>
  <li><a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">Language Models are Few-Shot Learners</a></li>
  </ol>
  <p>Additionally, if you have never seen transformers before, take a look at:</p>
  <ol>
  <li><a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention Is All You Need</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T03:53:11.140Z
Answers.LastActivityDate: 2020-11-30T03:53:11.140Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I am diving in data-to-text generation for long articles (&gt; 1000 words). After creating a template and fill it with data I am currently going down on paragraph level and adding different paragraphs, which are randomly selected and put together. I also added on a word level different outputs for date, time and number formats.</p>
  <p>The challenge I see is, that when creating large amounts of such generated texts they become boring to read as the uniqueness for the reader goes down.</p>
  <p>Furthermore, I also think it's easy to detect that such texts have been autogenerated. However, I still have to validate this hypotheses.</p>
  <p>I was wondering if there is an even better method to bring in variability in such a text?</p>
  <p>Can you suggest any methods, papers, resources or share your experience within this field.</p>
  <p>I highly appreciate your replies!</p>
  
Tags: <natural-language-processing><text-generation><natural-language-generation>
Title: Making generated texts from "data-to-text" more variable
UserDisplayName: Carol.Kar
ThreadText: # Making generated texts from "data-to-text" more variable
  
  (Asked by: username_0 on 2020-11-22)
  
  I am diving in data-to-text generation for long articles (> 1000 words). After creating a template and fill it with data I am currently going down on paragraph level and adding different paragraphs, which are randomly selected and put together. I also added on a word level different outputs for date, time and number formats.
  
  The challenge I see is, that when creating large amounts of such generated texts they become boring to read as the uniqueness for the reader goes down.
  
  Furthermore, I also think it's easy to detect that such texts have been autogenerated. However, I still have to validate this hypotheses.
  
  I was wondering if there is an even better method to bring in variability in such a text?
  
  Can you suggest any methods, papers, resources or share your experience within this field.
  
  I highly appreciate your replies!
  
  **Tags:** natural-language-processing, text-generation, natural-language-generation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-20. Score: 1 | Accepted answer
  
  You could handwrite different templates and choose probabilistically, according to writing style or pragmatic effects like irony and so on, but that very much depends on the domain.
  If you have tabular data, from which you want to generate text, you should probably forget about GPT and so on. You only have few control (despite copy mechanism) over the generation process, you actually predict the next most probable word sequence for a given length - GPTs don't author coherent text across paragraphs, especially not when text length is more than a few hundred words.
  
  Check the linguistic counterpart (<https://arxiv.org/abs/1703.09902>) to end2end generation systems. Breaking up the networks, pipelining again and using networks for controllable tasks.
  e.g. build a module that selects which attributes to produce. Create RDF triples from the column head words and values in your data base. Take a Text to text Transfer model (Google's T5) and transform into surface text. You should also have a look into the webNLG challenge (<https://webnlg-challenge.loria.fr/challenge_2020/>). This might help. Much of this is still open research. I am quite busy in this topic, so feel free to ask.
  
  > **Comments:**
  > **username_0**: Thx for your reply! I am also a researcher. Is there a way to get in touch with you?  
  > **username_1**: VIa Mail? Don't know whether stackexchange offers private messages, I think these comments will soon get deleted:)  
  > **username_0**: Btw if you are interested for an exchange message me on maggregor.samsa@gmail.com  
  
  ---
  
  ### Answer by username_2 on 2020-11-30. Score: 1
  
  The state of the art in text generation is the GPT model. [GPT-3](https://en.wikipedia.org/wiki/GPT-3), which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. [This](https://www.technologyreview.com/2020/07/20/1005454/openai-machine-learning-language-generator-gpt-3-nlp/) article and [this](https://www.technologyreview.com/2020/10/08/1009845/a-gpt-3-bot-posted-comments-on-reddit-for-a-week-and-no-one-noticed/) one should give you an example of how powerful it is at text generation.
  
  GPT is a [transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) based architecture, somewhat like [BERT](https://arxiv.org/abs/1810.04805). The main difference is it only takes into account left context, which is why it is so well suited for text generation.
  
  GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.
  
  Here is a [link](https://minimaxir.com/2019/09/howto-gpt2/) to a tutorial explaining the basics to get you started with GPT-2.
  
  If you are interested in diving into the relevant research papers:
  
  Here are the open AI papers on GPT 1-3
  
  1. [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
  2. [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
  3. [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
  
  Additionally, if you have never seen transformers before, take a look at:
  
  1. [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
  
  
  ---
  
  