STYLE EXAMPLE-GUIDED TEXT GENERATION USING GENERATIVE ADVERSARIAL TRANSFORMERS
We introduce a language generative model framework for generating a styled paragraph based on a context sentence and a style reference example. The framework consists of a style encoder and a texts decoder. The style encoder extracts a style code from the reference example, and the text decoder generates texts based on the style code and the context. We propose a novel objective function to train our framework. We also investigate different network design choices. We conduct extensive experimental validation with comparison to strong baselines to validate the effectiveness of the proposed framework using a newly collected dataset with diverse text styles. Both code and dataset will be released upon publication.
The paper introduces a model for generating text, conditionally on a context content and on a style information. Given a context sentence or paragraph, the task consists in generating plausible text with a content corresponding to this initial context and a style corresponding to another piece of text. The model is based on a transformer architecture inspired from GPT2. It is trained using a combination of loss functions: log likelihood, distillation w.r.t a pre-trained GPT2, style and adversarial loss. The model is evaluated on aggregations of multiple text corpora building a multi-style corpus. The proposed model combines several interesting features like making use of large pre-trained language models to guide the generation or the use of a latent GAN formulation for text decoding in order to avoid non differentiability. The authors propose different variants of the model for incorporating the style component into the transformer architecture. The role of the different loss components should however be better motivated. The _DIST and _GAN components are not directly linked to the objective and their importance for the model should be better stressed. The global model is probably quite heavy and a discussion on its complexity and on the training strategy would be useful in order to complete the description. The evaluation is performed on two corpora assembled by the authors and representing a variety of styles. This contribution could be helpful to the community. The authors introduce a series of evaluation criteria: perplexity of the generated text, style classifier, diversity and novelty. They also provide human evaluation. According to the quantitative criteria, one of the variants seems to perform reasonably well compared to the baselines. However, the evaluation is not convincing. It requires to be pushed further and the results and the system behavior should be better analyzed. The different examples of generated text are not consistent, and rather look like a random sampling over the vocabulary. This is in contradiction with the scores appearing in the tables and on the figures. One of the Baselines (SC) generates much more coherent texts – at least in the examples provided by the authors – while its perplexity score is similar to the authors’ model. The ablation study does not show the importance of the different loss components. For example, the models without the GAN or the DIST loss terms behave as well as the complete model does. Concerning the human evaluation, I do not understand the numbers in table 3-right: it seems that the model performs better at generating among 21 styles (-category) than it does with 3 styles? Overall, the paper represents a large amount of experimental work, but deserves a more in depth discussion of the usefulness of the different components. The experimental evaluation is still preliminary.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
STYLE EXAMPLE-GUIDED TEXT GENERATION USING GENERATIVE ADVERSARIAL TRANSFORMERS
We introduce a language generative model framework for generating a styled paragraph based on a context sentence and a style reference example. The framework consists of a style encoder and a texts decoder. The style encoder extracts a style code from the reference example, and the text decoder generates texts based on the style code and the context. We propose a novel objective function to train our framework. We also investigate different network design choices. We conduct extensive experimental validation with comparison to strong baselines to validate the effectiveness of the proposed framework using a newly collected dataset with diverse text styles. Both code and dataset will be released upon publication.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper introduces a model for generating text, conditionally on a context content and on a style information. Given a context sentence or paragraph, the task consists in generating plausible text with a content corresponding to this initial context and a style corresponding to another piece of text. The model is based on a transformer architecture inspired from GPT2. It is trained using a combination of loss functions: log likelihood, distillation w.r.t a pre-trained GPT2, style and adversarial loss. The model is evaluated on aggregations of multiple text corpora building a multi-style corpus. The proposed model combines several interesting features like making use of large pre-trained language models to guide the generation or the use of a latent GAN formulation for text decoding in order to avoid non differentiability. The authors propose different variants of the model for incorporating the style component into the transformer architecture. The role of the different loss components should however be better motivated. The _DIST and _GAN components are not directly linked to the objective and their importance for the model should be better stressed. The global model is probably quite heavy and a discussion on its complexity and on the training strategy would be useful in order to complete the description. The evaluation is performed on two corpora assembled by the authors and representing a variety of styles. This contribution could be helpful to the community. The authors introduce a series of evaluation criteria: perplexity of the generated text, style classifier, diversity and novelty. They also provide human evaluation. According to the quantitative criteria, one of the variants seems to perform reasonably well compared to the baselines. However, the evaluation is not convincing. It requires to be pushed further and the results and the system behavior should be better analyzed. The different examples of generated text are not consistent, and rather look like a random sampling over the vocabulary. This is in contradiction with the scores appearing in the tables and on the figures. One of the Baselines (SC) generates much more coherent texts – at least in the examples provided by the authors – while its perplexity score is similar to the authors’ model. The ablation study does not show the importance of the different loss components. For example, the models without the GAN or the DIST loss terms behave as well as the complete model does. Concerning the human evaluation, I do not understand the numbers in table 3-right: it seems that the model performs better at generating among 21 styles (-category) than it does with 3 styles? Overall, the paper represents a large amount of experimental work, but deserves a more in depth discussion of the usefulness of the different components. The experimental evaluation is still preliminary.
assistant
The paper introduces a model for generating text, conditionally on a context content and on a style information. Given a context sentence or paragraph, the task consists in generating plausible text with a content corresponding to this initial context and a style corresponding to another piece of text. The model is based on a transformer architecture inspired from GPT2. It is trained using a combination of loss functions: log likelihood, distillation w.r.t a pre-trained GPT2, style and adversarial loss. The model is evaluated on aggregations of multiple text corpora building a multi-style corpus. The proposed model combines several interesting features like making use of large pre-trained language models to guide the generation or the use of a latent GAN formulation for text decoding in order to avoid non differentiability. The authors propose different variants of the model for incorporating the style component into the transformer architecture. The role of the different loss components should however be better motivated. The _DIST and _GAN components are not directly linked to the objective and their importance for the model should be better stressed. The global model is probably quite heavy and a discussion on its complexity and on the training strategy would be useful in order to complete the description. The evaluation is performed on two corpora assembled by the authors and representing a variety of styles. This contribution could be helpful to the community. The authors introduce a series of evaluation criteria: perplexity of the generated text, style classifier, diversity and novelty. They also provide human evaluation. According to the quantitative criteria, one of the variants seems to perform reasonably well compared to the baselines. However, the evaluation is not convincing. It requires to be pushed further and the results and the system behavior should be better analyzed. The different examples of generated text are not consistent, and rather look like a random sampling over the vocabulary. This is in contradiction with the scores appearing in the tables and on the figures. One of the Baselines (SC) generates much more coherent texts – at least in the examples provided by the authors – while its perplexity score is similar to the authors’ model. The ablation study does not show the importance of the different loss components. For example, the models without the GAN or the DIST loss terms behave as well as the complete model does. Concerning the human evaluation, I do not understand the numbers in table 3-right: it seems that the model performs better at generating among 21 styles (-category) than it does with 3 styles? Overall, the paper represents a large amount of experimental work, but deserves a more in depth discussion of the usefulness of the different components. The experimental evaluation is still preliminary.