Auto-regressive text generation models usually focus on local fluency, and may cause inconsistent semantic meaning in long text generation. Further, automatically generating words with similar semantics is challenging, and hand-crafted linguistic rules are difficult to apply. We consider a text planning scheme and present a model-based imitation-learning approach to alleviate the aforementioned issues. Specifically, we propose a novel guider network to focus on the generative process over a longer horizon, which can assist next-word prediction and provide intermediate rewards for generator optimization. Extensive experiments demonstrate that the proposed method leads to improved performance. Text generation is an important area of investigation within machine learning. Recent work has shown excellent performance on a number of tasks, by combining reinforcement learning (RL) and generative models. Example applications include image captioning (Ren et al., 2017; Rennie et al., 2016), text summarization (Li et al., 2018b; Paulus et al., 2017; Rush et al., 2015), and adversarial text generation (Guo et al., 2017; Lin et al., 2017; Yu et al., 2017; Zhang et al., 2017; Zhu et al., 2018). The sequence-to-sequence framework (Seq2Seq) (Sutskever et al., 2014) is a popular technique for text generation. However, models from such a setup are typically trained to predict the next token given previous ground-truth tokens as input, causing what is termed exposure bias (Ranzato et al., 2016). By contrast, sequence-level training with RL provides an effective means of solving this challenge, by treating text generation as a sequential decision-making problem. By directly optimizing an evaluation score (cumulative rewards) (Ranzato et al., 2016), state-of-the-art results have been obtained in many text-generation tasks (Paulus et al., 2017; Rennie et al., 2016). However, one problem in such a framework is that rewards in RL training are particularly sparse, since a scalar reward is typically only available after an entire sequence has been generated. Furthermore, the recurrent models focus more on local fluency, and may cause inconsistent semantic meanings for long text generation. For RL-based text generation, most existing works rely on a model-free framework, which has been criticized for its high variance and poor sample efficiency (Sutton and Barto, 1998). On the other hand, while model-based RL methods do not suffer from these issues, they are usually difficult to train in complex environments. Further, a learned policy is usually restricted by the capacity of an environment model. Recent developments on model-based RL (Gu et al., 2016; Kurutach et al., 2018; Nagabandi et al., 2017) combine the advantages of these two approaches, and have achieved improved performance by learning a model-free policy, assisted by an environment model. In addition, model-based RL has been employed recently to solve problems with extremely sparse rewards, with curiosity-driven methods (Pathak et al., 2017). In this paper, we propose a model-based imitation-learning method to overcome the aforementioned issues in text-generation tasks. Our main idea is to employ an explicit guider network to model the generation environment in the feature space of sentence tokens, used to emit intermediate rewards by matching the predicted features from the guider network and features from generated sentences. The guider network is trained to encode global structural information of training sentences, and thus is useful to guide next-token prediction in the generative process. Within the proposed framework, to assist the guider network, we also develop a new type of self-attention mechanism to provide high-level planning-ahead information and maintain consistent semantic meaning. Text Generation Model Text generation models learn to generate a sentence Y = ( y 1 , . . . , y T ) of length T , possibly conditioned on some context X . Here each y t is a token from vocabulary A . Starting from the initial state s 0 , a recurrent neural network (RNN) produces a sequence of states ( s 1 , . . . , s T ) given an input sentence-feature representation ( e ( y 1 ) , . . . , e ( y T )) , where e (  ) denotes a word embedding function mapping a token to its d dimensional feature representation. The states are recursively updated with a function known as the cell : s t = h ( s t  1 , e ( y t )) . One typically assigns the following probability to an observation y at location t : p ( y | Y <t ) = [ softmax ( g ( s t ))] y . Together ( g, h ) specifies a probabilistic model  , i.e. , log  ( Y ) = (cid:88) t log p ( y t | Y <t ) . To train the model  , one typically uses maximum likelihood estimation (MLE), via minimizing the cross-entropy loss, i.e. , JMLE (  ) =  E [log  ( Y )] . In order to generate sentence Y s from a (trained) model, one iteratively applies the following operations: y st +1  Multi (1 , softmax ( g ( s t ))) , (2) s t = h ( s t  1 , e ( y st )) , (3) where Multi (1 ,  ) denotes one draw from a multinomial distribution. Model-Based Imitation Learning Text generation can be considered as an RL problem with a large number of discrete actions, deterministic transitions, and deterministic terminal rewards. It can be formulated as a Markov decision process (MDP) M = (cid:104)S , A , P, r,  (cid:105) , where S is the state space, A is the action space, P is the deterministic environment dynamics, r ( s , y ) is a reward function, and   (0 , 1) is the discrete-time discount factor. The policy   , parameterized by  , maps each state s  S to a probability distribution over A . The objective is to maximize the expected reward: J (  ) =  (cid:88) t =1 E P, (cid:2)  t  1  r ( s t , y t ) (cid:3) . In model-based imitation learning (Baram et al., 2017; Cheng et al., 2019), a model is built to make predictions for future state s t + (cid:52) t conditioned on the current state 1 , which can be used for action selection, e.g. , next-token generation. This model is typically a discrete-time system, taking the current state-action pair ( s t , y t ) as input, and outputting an estimate of the future state s t + (cid:52) t at time t + (cid:52) t . At each step t , y t is chosen based on the model, and the model will re-plan with the updated information from the dynamics. This control scheme is different from a standard model-based method, and is referred to as model-predictive control (MPC) (Nagabandi et al., 2017). Note that in our setting, the state in RL typically corresponds to the current generated sentences Y 1 ,...,t instead of the RNN state of generator (decoder). The model is illustrated in Figure 1, with an au-toeocoder (AE) structure for sentence feature extraction and generation. The encoder is shared for sentences from both training data and generated data, as explained in detail below. Overall, text generation can be formulated as an imitation-learning problem. At each timestep t , the agent, also called a generator (which corresponds to the LSTM decoder), takes the current LSTM state as input, denoted as s t . The policy   ( | s t ) parameterized by  is a conditional generator, to generate the next token (action) given s t , the observation representing the current generated sentence. The objective of text generation is to maximize the total reward as in (4). We detail the components for our proposed model in the following subsections. The guider network, implemented as an RNN with LSTM units, is adopted to model environment dynamics to assist text generation. The idea is to train a guider network such that its predicted sentence features at each time step are used to assist next-word generation and construct intermediate rewards, which in turn are used to optimize the sentence generator. Denote the guider network as G  ( s Gt  1 , f t ) , with parameters  and input arguments ( s Gt  1 , f t ) at time t , to explicitly write out the dependency on the guider network latent state s Gt  1 from the previous time step. Here f t is the input to the LSTM guider, which represents the feature of the current generated sentence extracted 1 (cid:52) t > 1 ; the model predicts future states based on the collected trajectories. by an encoder network. Specifically, let the current generated sentence be Y 1 ...t (encouraged to be the same as parts of a training sentence in train-ing), with f t calculated as: f t = Enc ( Y 1 ...t ) . The initial state of the guider network is the encoded feature of a true input sentence by the same convolutional neural network (CNN), i.e. , s G 0 = Enc ( X ) , where Enc (  ) denotes the encoder transformation, implemented with a CNN (Zhang et al., 2017). Importantly, the input to the guider network, at each time point, is defined by features from the entire sentence generated to that point. This provides an important guide to the LSTM decoder, accounting for the global properties of the generated text. Text Generation with Planning We first explain how one uses the guider network to guide next-word generation for the generator (the LSTM decoder in Figure 1). Our framework is inspired by the MPC method (Nagabandi et al., 2017), and can be regarded as a type of plan-ahead attention mechanism. Given the feature f t at time t from the current input sentence, the guider network produces a prediction G  ( s Gt  1 , f t ) as a future feature representation, by feeding f t into the LSTM guider. Since the training of the guider network is based on real data (detailed in the next paragraph), the predicted feature contains global-structure information of the training sentences. To utilize such information to predict the next word, we combine the predicted feature with the output of the decoder by constructing an attention-like mechanism. Specifi-cally, we first apply a linear transformation  on the predicted feature G  ( s Gt  1 , f t ) , forming a weight vector w t (cid:44)  (cid:0) G  ( s Gt  1 , f t ) (cid:1) . The weight w t is applied to the output O t of the LSTM decoder by an element-wise multiplication operation. The result is then fed into a softmax layer to generate the next token y t . Formally, the generative process O t = g ( s t  1 ) , w t =  ( G  ( s Gt  1 , f t )) , (5) y t  Multi (1 , softmax ( O t  w t )) , (6) s Gt = h G ( s Gt  1 , f t ) , s t = h ( s t  1 , e ( y t )) . (7) Guider Network Training Given a sentence of feature representations ( f 1 , f 2 , . . . f T ) for a training sentence, we seek to update the guider network such that it is able to predict f t + c given f t , where c > 0 is the number of steps that are looked ahead. We implement this by forcing the predicted feature, G  ( s Gt , f t ) , to match both the sentence feature f t + c (first term in (8)) and the corresponding feature-changing direction (second term in (8)). This is formalized by maximizing an objective function of the following form at time t : where D cos (  ,  ) denotes the cosine similarity 2 . By maximizing (8), an ideal guider network should be able to predict the true next words conditioned on the current word in a sentence. As a result, the prediction is used to construct an intermediate reward, used to update the generator (the LSTM decoder), as described further below. As in many RL-based text-generation methods, such as SeqGAN (Yu et al., 2017) and LeakGAN (Guo et al., 2017), the generator is updated based on policy-gradient methods. As a result, collecting rewards in the generation process is critical. Though SeqGAN (Yu et al., 2017) has proposed to use rollout to get rewards for each generated word, the variance of the rewards is typically too high to be useful practically. In addition, the computational cost may be too high for practical use. We below describe how to use the proposed guider network to define intermediate rewards, leading to a definition of feature-matching reward. Feature-Matching Rewards We first define an intermediate reward to generate a particular word. The idea is to match the ground-truth features from the CNN encoder in Figure 1 with those generated from the guider network. Equation (8) indicates that the further the generated feature is from the true feature, the smaller the reward should be. To this end, for each time t , we define the intermediate reward for generating the current word as: r gt = 1 2 c c (cid:88) i =1 ( D cos ( f t ,  f t )+ D cos ( f t  f t  i ,  f t  f t  i )) , where  f t = G  ( s Gt  c  1 , f t  c ) is the predicted feature. Intuitively, f t  f t  i measures the difference between the generated sentences in feature space; the reward is high if it matches the predicted feature transition  f t  f t  i from the guider network. At the last step of text generation, i.e. , t = T , the corresponding reward measures the quality of the whole generated sentence, thus it is called a final reward. The final reward is defined differently from the intermediate reward, discussed below for both the unconditionaland conditional-generation cases. Note that a token generated at time t will influence not only the rewards received at that time but also the rewards at subsequent time steps. Thus we propose to define the cumulative reward, (cid:80) Ti = t  i r gi with  a discount factor, as a feature-matching reward . Intuitively, this encourages the generator to focus on achieving higher long-term rewards. Finally, in order to apply policy gradient to update the generator, we combine the feature-matching reward with the problem-specific final reward, to form a Q -value reward specified below. Similar to SeqGAN, the final reward is defined as the output of a discriminator, evaluating the quality of the whole generated sentence, i.e. , the smaller the output, the less likely the generation is a true sentence. As a result, we combine the adversarial reward r f  [0 , 1] by the discriminator (Yu et al., Generator Optimization The generator is initialized by pre-training on sentences with an autoencoder structure, based on MLE training. After that, the final Q -value reward Q t is used as a reward for each time t , with standard policy gradient optimization methods to update the generator. Specifically, the policy gradient is J = E ( s t  1 ,y t )    [ Q t   log p ( y t | s t  1 ; ,  )]   J = E ( s t  1 ,y t )    [ Q t   log p ( y t | s t  1 ; ,  )] where p ( y t | s t  1 ; ,  ) is the probability of generating y t given s t  1 in the generator. Algorithm 1 describes the proposed model-based imitation learning framework for text generation. Model-based or Model-free Text generation seeks to generate the next word (action) given the current (sub-)sentence (state). The generator is considered as an agent that learns a policy to predict the next word given its current state. In previous work (Ranzato et al., 2016), a metric reward is given and the generator is trained to only maximize the metric reward by trial, thus this is model-free learning. In the proposed method, the guider network models the environment dynamics, and is trained by minimizing the cosine similarity between the prediction and the ground truth on real text. For generator training, it maximizes the reward which is determined by the metric and guider network, and thus is model-free learning with model-based boosting (Gu et al., 2016). The model predictive control scheme is included in our method, where the guider network is used to help next-word selection at each time-step. As illustrated in Figure 2, our framework naturally provides a way for style transfer, where the guider network plays the role of style selection, and the generator only focuses on maintaining content without considering the styles. To make the guider network focus on the guidance of styles, we assign the label l as the initial state s G 0 of the guider network. Specifically, at each step t , we feed the current sentence representation f t and label l into the guider network: O t = g ( s t  1 ) , w t =  ( G  ( s Gt  1 , [ f t , l ])) , (9) y t  Multi (1 , softmax ( O t  w t )) . For the generator, we put an adversarial regularizer on the encoded latent s 0 ( X ) and penalize it if it contains the sentiment information, by maximizing the entropy, i.e., max (cid:80) l p ( l | s 0 ( X )) log p ( l | s 0 ( X )) , where p is a pre-trained classifier. Intuitively, the generator gives candidate words represented by O t , while the guider makes a choice implicitly by w t based on the sentiment information. The sentiment information is contained in w t , while the content of the original sentence is represented by O t . To achieve style-transfer, one feeds the original sentence X with the target style label l to get the transferred sentence Y with style l . Following previous work (Hu et al., 2017; Yang et al., 2018; Cheng et al., 2020), we adopt a classifier as the discriminator and the soft-argmax approach (Kusner and Miguel, 2016) for the update of generator instead of policy gradient (Sutton and Barto, 1998). We first review related works that combine RL and GAN for text generation. As one of the most representative models in this direction, SeqGAN (Yu et al., 2017) adopts Monte-Carlo search to calculate rewards. However, such a method introduces high variance in policy optimization. There are a number of works proposed subsequently to improve the reward-generation process. For example, RankGAN (Lin et al., 2017) proposes to replace the reward from the GAN discriminator with a ranking-based reward, MaliGAN (Che et al., 2017) modifies the GAN objective and proposes techniques to reduce gradient variance, MaskGAN (Fedus et al., 2018) uses a filling technique to define a Q -value reward for sentence completion, RelGAN (Nie et al., 2019) uses a relational memory based generator for the long-distance dependency modeling, FM-GAN (Chen et al., 2018) uses a feature mover distance to match features of real and generated sentences inspired by optimal transport (Chen et al., 2019; Zhang et al., 2018), and LeakGAN (Guo et al., 2017) tries to address the sparse-reward issue for long-text generation with hierarchical RL by utilizing the leaked information from a GAN discriminator. One problem of LeakGAN is that it tends to overfit the training data, yielding generated sentences that are often not diverse. By contrast, by relying on a model-based imitation learning approach, our method learns global-structure information, which generates more-diverse sentences, and can be extended to conditional text generation. Zhang et al. (2020) designed a differentiable nested Wasserstein distance for semantic matching, which can be applied for further improvement. RL techniques can also be used in other ways for text generation (Bachman and Precup, 2015). For example, Ranzato et al. (2016) trained a Seq2Seq model by directly optimizing the BLEU/ROUGE scores with the REINFORCE algorithm. To reduce variance of the vanilla REINFORCE, Bahdanau et al. (2017) adopted the actor-critic framework for sequence prediction. Furthermore, Rennie et al. (2016) trained a baseline algorithm with a greedy decoding scheme for the REINFORCE method. Note that all these methods can only obtain reward after a whole sentence is generated. Planning techniques in RL have also been explored to improve text generation (Gulcehre et al., 2017; Serdyuk et al., 2018). Zhang et al. (2020) introduced the self-imitation scheme to exploit historical high-quality sentences for enhanced exploration. Compared to these related works, the proposed guider network can provide a planning mechanism and intermediate rewards. We test the proposed framework on unconditional and conditional text generation tasks, and analyze the results to understand the performance gained by the guider network. We also perform an ablation investigation on the improvements brought by each part of our proposed method, and consider non-parallel style transfer. All experiments are conducted on a single Tesla P100 GPU and implemented with TensorFlow and Theano. Details of the datasets, the experimental setup and model architectures are provided in the Appendix. Encoder as the feature extractor For unconditional generation, the feature extractor that generates inputs for the guider network shares the CNN part of the encoder. We stop gradients from the guider network to the encoder CNN in the training process. For conditional generation, we use a pretrained feature extractor, trained similarly to the unconditional generation. Training procedure As with many imitation-learning models (Bahdanau et al., 2017; Rennie et al., 2016; Sutskever et al., 2014), we first train the encoder-decoder part based on the off-policy data with an MLE loss. Then we use RL training to fine-tune the trained generator. We adaptively transfer the training from MLE loss to RL loss, similar to (Paulus et al., 2017; Ranzato et al., 2016). Initial states We use the same initial state for both the generator and the guider networks. For conditional generation, the initial state is the encoded latent code of the conditional information for both training and testing. For unconditional generation, the initial state is the encoded latent code of a target sentence in training and random noise in testing. We focus on adversarial text generation, and compare our approach with a number of related works (Guo et al., 2017; Lin et al., 2017; Yu et al., 2017; Zhang et al., 2017; Zhu et al., 2018). In this setting, a discriminator in the GAN framework is added to the model in Figure 1 to guide the generator to generate high-quality sentences. This is implemented by defining the final reward to be the output of the discriminator. All baseline experiments are implemented on the texygen platform (Zhu et al., 2018). We adopt the BLEU score, referenced by the test set (test-BLEU, higher value implies better quality) and itself (self-BLEU, lower value implies better diversity) (Zhu et al., 2018) to evaluate quality of generated samples, where test-BLEU evaluates the reality of generated samples, and self-BLEU measures the diversity. A good generator should achieve both a high test-BLEU score and a low self-BLEU score. In practice, we use (cid:52) t = c = 4 and  = 0 . 25 . We call the proposed method guider-matching GAN (GMGAN) for unconditional text generation. Short Text Generation: COCO Image Captions We use the COCO Image Captions Dataset, in which most sentences have a length of about 10 words. Since we consider unconditional text generation, only image captions are used as the training data. After preprocessing, we use 120,000 random sample sentences as the training set, and 10,000 as the test set. The BLEU scores with different methods are listed in Table 1. We observe that GMGAN performs significantly better than the baseline models. Specifically, besides achieving higher test-BLEU scores, the proposed method also generates samples with very good diversity in terms of self-BLEU scores. LeakGAN represents the state-of-the-art in adversarial text generation, however, its diversity measurement is relatively poor (Zhu et al., 2018). We suspect that the high BLEU score achieved by LeakGAN is due to its mode collapse on some good samples, resulting in high self-BLEU scores. Other baselines achieve lower self-BLEU scores since they cannot generate reasonable sentences. Long Text Generation: EMNLP2017 WMT Following (Zhu et al., 2018), we use the News section in the EMNLP2017 WMT4 Dataset as our training data. The dataset consists of 646,459 words and 397,726 sentences. After preprocessing, the training dataset contains 5,728 words and 278,686 sentences. The BLEU scores with different methods are provided in Table 2. Compared with other methods, LeakGAN and GMGAN achieve comparable test-BLEU scores, demonstrating high-quality generated sentences. Again, LeakGAN tends to over-fit on training data, leading to much higher (worse) self-BLEU scores. Our proposed GMGAN shows good diversity of long text generation with lower self-BLEU scores. Other baselines obtain both low self-BLEU and test-BLEU scores, leading to more random generations. Human Evaluation Simply relying on the above metrics is not sufficient to evaluate the proposed method (Caccia et al., 2018). Following previous work (Guo et al., 2017), we perform human evaluations using Amazon Mechnical Turk, evaluating the text quality based on readability and meaningfulness (whether sentences make sense) on the EMNNLP2017 WMT News dataset. We ask the worker to rate the input sentence with scores scal-Scores Criteria 5 (Best) It is consistent, informative, grammatically correct. methods on EMNLP2017 WMT dataset. ing from 1 to 5, with 1 as the worst score and 5 as the best. The detailed criteria is listed in Table 3. We require all the workers to be native English speakers, with approval rate higher than 90% and at least 100 assignments completed. We randomly sample 100 sentences generated by each model. Ten native English speakers on Amazon Mechanical Turk are asked to rate each sentence. The average human rating scores are shown in Table 4, indicating GMGAN achieves higher human scores compared to other methods. As examples, Table 5 illustrates some generated samples by GMGAN and its baselines. The performance on the two datasets indicates that the generated sentences of GMGAN are of higher global consistency and better readability than SeqGAN and LeakGAN. More generated examples are provided in the Appendix. Ablation Study We conduct ablation studies on long text generation to investigate the improvements brought by each part of our proposed method. We first test the benefits of using the guider network. Among the methods compared, Guider is the standard MLE model with the guider network. We further compare RL training with i ) only final rewards , ii ) only feature-matching rewards, and iii ) combining both rewards, namely GMGAN. The results are shown in Table 6. We observe that guider network plays an important role in improving the performance. RL training with final rewards given by a discriminator typically damages the generation quality, but feature-matching reward produces sentences with much better diversity due to the ability of exploration. Case Study of Guider-Matching Rewards Figure 3(a) illustrates the feature-matching rewards in the generation. Figure 3(a) shows an example of failure generation in the training stage, when two sentences are combined by the word  was '. It is grammatically wrong to select  was ' for the generator, thus the guider network gives a small reward. We can see that the rewards become lower with more time steps, which is consistent with the exposure bias. Figure 3(b) shows a successful generation, where the rewards given by the guider are relatively high (larger than 0.5). These observations validate that: ( i ) exposure bias exists in MLE training. ( ii ) RL training with exploration can help reduce the effects of exposure bias. ( iii ) Our proposed feature-matching rewards can provide meaningful guidance to maintain sentence structure and fluency. We test the proposed framework on the non-parallel text-style-transfer task, where the goal is to transfer one sentence in one style ( e.g. , positive) to a similar sentence but with a different style ( e.g. , negative). Pair-wise information should be inferred from the training data, which becomes more challenging. For a fair comparison, we use the same data and its split method as in (Shen et al., 2017). Specifically, there are 444,000, 63,500, and 127,000 sentences with either positive or negative sentiments in the training, validation and test sets, respectively. To measure whether the original sentences (in the test set) have been transferred to the desired sentiment, we follow the settings of (Shen et al., 2017) and employ a pretrained CNN classifier, which achieves an accuracy of 97 . 4% on the validation set, to evaluate the transferred sentences. We also report the BLEU scores with original sentences (BLEU) and human references (BLEU-ref) (Li et al., 2018a), to evaluate the content preservation of transferred sentences. Results are summarized in Table 7. Our proposed model exhibits higher transfer accuracy and better content preservation, indicating the guider network provides good sentiment guidance to better preserve the content information. We have proposed a model-based imitation-learning framework for adversarial text generation, by introducing a guider network to model the generation environment. The guider network provides a plan-ahead mechanism for next-word selection. Furthermore, this framework can alleviate the sparse-reward issue, as the intermediate rewards are used to optimize the generator. Our proposed models are validated on both unconditional and conditional text generation, including adversarial text generation and non-parallel style transfer. We achieve improved performance in terms of generation quality and diversity for unconditional and conditional generation tasks. Acknowledgement The authors would like to thank the anonymous reviewers for their insightful comments. The research was supported in part by DARPA, DOE, NIH, NSF and ONR.
abstain abstain method objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain objective other other other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other abstain abstain abstain other abstain other other abstain abstain abstain abstain other method abstain abstain method abstain abstain abstain method objective other other other other other objective other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain objective abstain abstain objective result abstain abstain