Yoshimasa Tsuruoka The University of Tokyo tsuruoka@logos.t.u-tokyo.ac.jp Abstract A major obstacle in reinforcement learning-based sentence generation is the large action space whose size is equal to the vocabulary size of the target-side language. To improve the efficiency of reinforcement learning, we present a novel approach for reducing the action space based on dynamic vocabulary prediction. Our method first predicts a fixed-size small vocabulary for each input to generate its target sentence. The input-specific vocabularies are then used at supervised and reinforcement learning steps, and also at test time. In our experiments on six machine translation and two image captioning datasets, our method achieves faster reinforcement learning (  2.7x faster) with less GPU memory (  2.3x less) than the full-vocabulary counterpart. We also show that our method more effectively receives rewards with fewer iterations of supervised pre-training. Sentence generation with neural networks plays a key role in many language processing tasks, including machine translation (Sutskever et al., 2014), image captioning (Lin et al., 2014), and abstractive summarization (Rush et al., 2015). The most common approach for learning the sentence generation models is maximizing the likelihood of the model on the gold-standard target sentences. Recently, approaches based on reinforcement learning have attracted increasing attention to reduce the gap between training and test situations and to directly incorporate task-specific and more flexible evaluation metrics such as BLEU scores (Papineni et al., 2002) into optimization (Ranzato et al., 2016). tionally demanding to be used with large training data. In reinforcement learning for sentence generation, selecting an action corresponds to selecting a word in the vocabulary V . The number of possible actions at each time step is thus equal to the vocabulary size, which often exceeds tens of thousands. Among such a large set of possible actions, at most N actions are selected if the length of the generated sentence is N , where we can assume N (cid:28) | V | . In other words, most of the possible actions are not selected, and the large action space slows down reinforcement learning and consumes a large amount of GPU memory. In this paper, we propose to accelerate reinforcement learning by reducing the large action space. The reduction of action space is achieved by predicting a small vocabulary for each source input. Our method first constructs the small input-specific vocabulary by selecting K (  1000) relevant words, and then the small vocabulary is used at both training and test time. Our experiments on six machine translation and two image captioning datasets show that our method enables faster reinforcement learning with less GPU memory than the standard full softmax method, without degrading the accuracy of the sentence generation tasks. Our method also works faster at test time, especially on CPUs. The implementation of our method is available at https: //github.com/hassyGo/NLG-RL . We first describe a neural machine translation model and an image captioning model as examples of sentence generation models. Machine translation is a text-to-text task, and image captioning is an image-to-text task. We then review how reinforcement learning is used, and present a simple and efficient method to accelerate the training. Recurrent Neural Networks (RNNs) are widely used to generate sentences by outputting words one by one (Sutskever et al., 2014). To generate a sentence Y = ( y 1 , y 2 , . . . , y N ) , where N is its length, given a source input X , a hidden state h t  R d is computed for each time step t (  1) by using its previous information: h t = RNN ( h t  1 , e ( y t  1 ) , s t  1 ) , (1) where RNN(  ) is an RNN function, e ( y t  1 )  R d is a word embedding of y t  1 , and s t  1  R d is a hidden state optionally used to explicitly incorporate the information about the source input X into the transition. We employ Long Short-Term Memory (LSTM) units (Hochreiter and Schmid-huber, 1997) for the RNN function. The task here is to predict the t -th word y t by computing a target word distribution p ( y | y <t , X )  R | V | , where | V | represents the vocabulary size of the target language. p ( y | y <t , X ) is used to generate a sentence by either greedy/beam search or random sampling. To learn the model parameters, the following cross entropy loss is usually employed: L c ( Y g , X ) =  N g (cid:88) t =1 log p ( y = y t | y <t , X ) , (2) where we assume that the target sentence Y g is the gold sequence. Once we train the model, we can use it to generate unseen sentences. Machine translation In the context of machine translation, the source input X corresponds to a source sentence ( x 1 , x 2 , . . . , x M ) of length M . Each word x i is also associated with a word embedding  e ( x i )  R d . We assume that a hidden state  h i  R 2 d is computed for each x i by using a bi-directional RNN with LSTM units (Graves and Schmidhuber, 2005). That is,  h i is the concatenation of x i 's d -dimensional hidden states [  h i ;  h i ] computed by a pair of forward and backward RNNs. We set the initial hidden state of the sentence generator as h 0 =  h M +  h 1 . Following an attention mechanism proposed in Luong et al. (2015), s t for predicting y t is computed as follows: s t = tanh (cid:32) W s (cid:34) h t ; M (cid:88) i =1 a i  h i (cid:35) + b s (cid:33) , (3) where a i = f ( h t , i,  h ) is the global-attention function in Luong et al. (2015), W s  R d  3 d is a weight matrix, and b s  R d is a bias vector. s t is then used to compute the target word distribution: p ( y | y <t , X ) = softmax( W p s t + b p ) , (4) where W p  R | V | d is a weight matrix, and b p  R | V | is a bias vector. Image captioning In the case of image captioning, the source input X corresponds to an image to be described. We assume that in our preprocessing step, each input image is fed into a convolutional neural network to extract its fixed-length feature vector f  R d f . More specifically, we use the pre-computed feature vectors provided by Kiros et al. (2014), and the feature vectors are never updated in any model training processes. The input feature vector is transformed into the initial hidden state h 0 = tanh ( W f f + b f ) , where W f  R d  d f is a weight matrix, and b f  R d is a bias vector. In contrast to machine translation, we do not use s t  1 in Equation (1); more concretely, we do not use any attention mechanisms for image captioning. Therefore, we directly use the hidden state h t to compute the target word distribution: p ( y | y <t , X ) = softmax( W p h t + b p ) , (5) where the weight and bias parameters are analogous to the ones in Equation (4). For both of the tasks, we use the weight-tying technique (Inan et al., 2017; Press and Wolf, 2017) by using W p as the word embedding matrix. That is, e ( y t ) is the y t -th row vector in W p , and the technique has shown to be effective in machine translation (Hashimoto and Tsuruoka, 2017) and text summarization (Paulus et al., 2018). One well-known limitation of using the cross entropy loss in Equation (2) is that the sentence generation models work differently at the training and test time. More concretely, the models only observe gold sequences at the training time, whereas the models have to handle unseen sequences to generate sentences at the test time. To bridge the gap, reinforcement learning has started gaining much attention (Ranzato et al., 2016; Wu et al., 2016; Rennie et al., 2017; Zhang and Lapata, 2017; Paulus et al., 2018; Yang et al., 2018). In this work, we focus on the most popular method called REINFORCE (Williams, 1992). 1 In REINFORCE, the sentence generation model sets an initial state given a source input, and then iterates an action selection and its corresponding state transition. The action selection corresponds to randomly sampling a target word from Equation (4) and (5), and the state transition corresponds to the RNN transition in Equation (1). Once a sentence is generated, an approximated loss function is defined as follows: L r ( Y, X ) =  N (cid:88) t =1 R t log p ( y = y t | y <t , X ) , (6) where R t is the reward at time step t , and the loss is approximated by the single example Y . R t is used to evaluate how good the t -th action selection is. Unlike maximum likelihood training, the reward function can be defined by using task-specific evaluation scores like BLEU for machine translation. In this paper, we employ GLEU proposed by Wu et al. (2016), a variant of sentence-level BLEU. Following the implementation in Ranzato et al. (2016), we define R t = GLEU( Y, Y g )  b t , where b t is a baseline value estimating the future reward from the next time step to reduce the variance of the gradients. To estimate b t , we jointly train a linear regression model by minimizing (cid:107) b t  GLEU( Y, Y g ) (cid:107) 2 , and b t is computed as b t =  ( W r  s t + b r ) , where W r  R d is a weight vector, b r is a bias,  (  ) is the logistic sigmoid function, and in the case of image captioning, h t is used instead of s t . Overall model training The reinforcement learning step is usually applied after pre-training the models with the cross entropy loss in Equation (2). At the REINFORCE phase, we define the following joint loss function: L = L c + (1   ) L r , (7) where  is a hyperparameter, and  = 0 . 0 usually leads to unstable training (Wu et al., 2016). The vocabulary size | V | is usually more than ten thousands for datasets covering many sentences with a variety of topics. However, for example, at most 100 unique words are selected when generating a sentence of length 100. That is, the output length N is much smaller than the vocabulary 1 We tried self critic (Rennie et al., 2017), but did not observe significant improvement over REINFORCE. size | V | , and this fact motivated us to reduce the large action space. Moreover, we have in practice found that REINFORCE runs several times slower than the supervised learning with the cross entropy loss. To accelerate the training, we propose to construct a small action space for each source input. In other words, our method selects a small vocabulary V (cid:48) of size K for each source input in advance to the model training. In this section, we assume that V (cid:48) is given and represented with a sparse binary matrix MX  RK | V | , where there are only K non-zero elements at position ( i, w i ) for 1  i  K . w i is a unique word index in V . MX is used to construct a small subset of the parameters in the softmax layer: W (cid:48) p = MXW p , b (cid:48) p = MX b p , (8) and W (cid:48) p  RK  d and b (cid:48) p  RK are used instead of W p and b p in Equation (4) and (5). Therefore, in mini-batched processes with a mini-batch size B , our method constructs B different sets of ( W (cid:48) p , b (cid:48) p ) . Relationship to previous work Sampling-based approximation methods have previously been studied to reduce the computational cost at the large softmax layer in probabilistic language modeling (Ji et al., 2016; Zoph et al., 2016), and such methods are also used to enable one to train neural machine translation models on CPUs (Eriguchi et al., 2016). The construction of ( W (cid:48) p , b (cid:48) p ) in our method is similar to these softmax approximation methods in that they also sample small vocabularies either at the word level (Ji et al., 2016), sentence level (Hashimoto and Tsuruoka, 2017), or mini-batch level (Zoph et al., 2016). However, one significant difference is that the approximation methods work only at training time using the cross entropy loss, and full softmax computations are still required at test time. The difference is crucial because a sentence generation model needs to simulate its test-time behavior in reinforcement learning. The remaining question is how to construct the input-specific vocabulary V (cid:48) for each source input X . This section describes our method to construct V (cid:48) by using a vocabulary prediction model which is separated from the sentence generation models. In the vocabulary prediction task, the input is the source X (source sentences or images) to be described, and the output is V (cid:48) . We should be careful not to make the prediction model computationally expensive; otherwise the computational efficiency by our method would be canceled out. To feed the information about X into our vocabulary prediction model, we define an input vector v ( X )  R d v . For image captioning, we use the feature vector f described in Section 2.1: v ( X ) = W v f + b v , where W v  R d v  d f is a weight matrix, and b v  R d v is a bias vector. For machine translation, we employ a bag-of-embeddings representation: v ( X ) = 1 M (cid:80) Mi =1  e v ( x i ) , where the d v -dimensional word embedding  e v ( x i )  R d v is different from  e ( x i ) used in the machine translation model. By using the different set of the model parameters, we avoid the situation that our vocabulary prediction model is affected during training the sentence generation models. Relationship to previous work Vocabulary prediction has gained attention for training sequence-to-sequence models with the cross entropy loss (Weng et al., 2017; Wu et al., 2017), but not for reinforcement learning. Compared to our method, previous methods jointly train a vocabulary predictor by directly using source encoders as input to the predictor. One may expect joint learning to improve both of the vocabulary predictor and the sentence generator, but in practice such positive effects are not clearly observed. Weng et al. (2017) reported that the joint learning improves the accuracy of their machine translation models, but our preliminary experiments did not indicate such accuracy gain. Such a joint training approach requires the model to continuously update the vocabulary predictor during REINFORCE, because the encoder is shared. That is, the action space for each input changes during reinforcement learning, and we observed unstable training. Therefore, this work separately models the vocabulary predictor and focuses on the effects of using the small vocabularies for REINFORCE. Another note is that Jean et al. (2015) and L'Hostis et al. (2016) also proposed to construct small vocabularies in advance to the cross entropy-based training. They suggest that the use of word alignment works well, but using the word alignment is not general enough, considering that there exist different types of source input. By contrast, our method can be straightforwardly applied to the two sentence generation tasks with the different input modalities (i.e. image and text). Once the input representation v ( X ) is computed, we further transform it by a single residual block (He et al., 2016): r ( X ) = Res ( v ( X ))  R d v . 2 Then r ( X ) is fed into a prediction layer: o =  ( W o r ( X ) + b o ) , (9) where W o  R | V | d v is a weight matrix, and b o  R | V | is a bias vector. The i -th element o i corresponds to the probability that the i -th word in the target vocabulary V appears in the target sentence Y given its source X . We use the training data for the sentence generations tasks to train the vocabulary predictor. For each X in the training data, we have its gold target sentence Y g . We train the vocabulary predictor as a multi-label classification model by the following loss function:  | V | (cid:88) i =1 ( t i log o i + (1  t i ) log(1  o i )) , (10) where t i is equal to 1 . 0 if the i -th word in V is included in Y g , and otherwise t i is 0 . 0 . In practice, we apply the label smoothing technique (Szegedy et al., 2016) to the loss function. We evaluate the accuracy of the vocabulary predictor by using a separate development split D : # of correctly predicted words in D # of words in D , (11) where we select the topK predictions in Equation (9) for each source input X in D , and the evaluation metric is a recall score. We use the topK words to construct the input-specific vocabularies V (cid:48) for the sentence generation models, and we restrict that the recall is 100% for the training data. We describe our experimental settings, and the details can be found in the supplemental material. 2 We can use arbitrary types of hidden layers or even linear models like SVMs, but we found this one performed the best. We describe the details of this in the supplemental material. We used machine translation datasets of four different language pairs: English-to-German (En-De), English-to-Japanese (En-Ja), English-to-Vietnamese (En-Vi), and Chinese-to-Japanese (Ch-Ja). For image captioning, we used two datasets: MS COCO (Lin et al., 2014) and Flickr8K. Table 1 summarizes the statistics of the training datasets, where the number of training examples (Size), the target vocabulary size ( | V | ), and the maximum length of the target sentences ( max( N ) ) are shown. For the machine translation datasets, we manually set max( N ) and omitted training examples which violate the constraints. En-De: We used 100,000 training sentence pairs from news commentary and newstest2015 as our development set, following Eriguchi et al. (2017). En-Ja: We used parallel sentences in ASPEC (Nakazawa et al., 2016) and constructed three types of datasets: En-Ja (100K), En-Ja (2M), and En-Ja (2M, SW). The 100K and 2M datasets were constructed with the first 100,000 and 2,000,000 sentence pairs, respectively. To test our method using subword units, we further preprocessed the 2M dataset by using the Sentence-Piece toolkit (Kudo and Richardson, 2018) to construct the En-Ja (2M, SW) dataset. En-Vi: We used the pre-processed datasets provided by Luong and Manning (2015). Our development dataset is the tst2012 dataset. Ch-Ja: We constructed the Ch-Ja dataset by using the first 100,000 sentences from ASPEC . MS COCO and Flickr8K: We used the preprocessed datasets provided by Kiros et al. (2014). We can also download the 4096-dimensional feature vectors f (i.e., d f = 4096 ). vocabulary predictor with a learning rate of 0 . 08 and a mini-batch size of 128. The model for each setting was tuned based on recall scores (with K = 1000 ) for the development split. We set d = 256 with single-layer LSTMs for all the experiments, except for the En-Ja (2M) and (2M, SW) datasets. For the larger En-Ja datasets, we set d = 512 with two-layer LSTMs. We used stochastic gradient decent with momentum, with a learning rate of 1 . 0 , a momentum rate of 0 . 75 , and a mini-batch size of 128. The model for each setting was tuned based on BLEU scores for the development split. All of the models achieved the best BLEU scores for all the datasets within 15 to 20 training epochs. Each of the selected models with the best BLEU scores was used for the following REINFORCE step. For REINFORCE, we set  = 0 . 005 , and the learning rate was set to 0 . 01 . The REINFORCE steps required around 5 epochs to significantly improve the BLEU scores. We used a single GPU of NVIDIA GeForce GTX 1080 3 to run experiments for the En-De, En-Ja (100K), En-Vi, Ch-Ja, MS COCO, and Flickr8K datasets. For the En-Ja (2M) and En-Ja (2M, SW) datasets, we used a single GPU of NVIDIA Tesla V100 4 to speedup our experiments. Mini-batch splitting It should be noted that our small softmax method can be run even on the single GTX 1080 GPU for the larger translation datasets, whereas the full softmax method runs out of the GPU memory. A typical strategy to address such out-of-memory issues is to use multiple GPUs, but we have found that we need at most eight GPUs to conduct our experiments on the full softmax method with REINFORCE. 5 Moreover, using the multiple GPUs does not always speedup the training time. We instead employ another strategy to split the mini-batch at each training iteration. First, we sort the mini-batch examples according to the lengths of the source (or target) text, and then split the mini-batch into S sets of the training examples. For example, in our case the 3 The GPU memory capacity is 11,178MiB. mini-batch size is 128, and if S is set to 4, each of the smaller sets includes 32 training examples. We perform back-propagation for each set one by one, and at each step we delete the corresponding computational graphs to reduce the GPU memory consumption. Finally, the accumulated partial derivatives are used to update the model parameters. More details can be found in our Pytorch 0.4 implementation. Figure 1 shows recall scores with respect to different values of the small vocabulary size K for each dataset. We can see that the recall scores reach 95% with K = 1000 for most of the datasets. One exception is the En-De dataset, and this is not surprising because a German vocabulary would become sparse by many compound nouns. These results show that our vocabulary predictor works well for source inputs of different modalities (text and image) and their corresponding different target languages. Our method also works at the subword level as well as at the standard word level. For training the sentence generation models, we set K = 500 for the Flickr8K dataset and K = 1000 for the other datasets. The goal of this paper is achieving efficient reinforcement learning for sentence generation to encourage future research, but before evaluating the efficiency of our method, we show that using the small vocabularies does not degrade the accuracy of the sentence generation models. Table 2 shows BLEU scores for the development splits of the four machine translation and two image captioning datasets. The BLEU scores are averaged over five different runs with different random seeds, and the standard deviations are also reported. We can see in Table 2 that our method (Small softmax) keeps the BLEU scores as high as those of Full softmax. For some datasets, the BLEU scores of our method are even better than those of the full softmax method. The trend is consistent in both of the cross entropy training phase and the REINFORCE phase. These results indicate that our method works well for different machine translation and image captioning datasets. We also confirmed that our experimental results are competitive with previously reported results when using the same training datasets; for example, our En-Vi test set result on tst2013 is 27.87  0.21 (cf. 26.9 in Luong and Manning (2015)). Better generation of rare words These BLEU scores suggest that our method for reinforcement learning has the potential to outperform the full softmax baseline. However, it is still unclear what is the potential advantage in terms of generation quality. We therefore analyzed the differences between output sentences of the small and full softmax methods, following Ott et al. (2018). Figure 2 shows the results of the En-De translation dataset, 0 1 2 3 4 5 6 10 20 30 40 50 60 70 80 90 100 O b s e r v ed f r equen cy [ % ] Frequency percentile in the training data Small softmax (K=1000) Full softmax Reference Figure 2: An analysis on the En-De translation results. and we observed the same trend for all the other datasets. Each entry is computed as follows: # of output words in each percentile # of output words , (12) where the 10 percentile includes the top 10% of the most frequent words, and the 100 percentile includes the top 10% of the most infrequent words. We can see that our small softmax method better outputs rare words, and these results suggest that using input-specific vocabularies is useful in controlling action spaces for reinforcement learning. Effectiveness with fewer pre-training steps We followed the standard practice that the models are pre-trained by maximum likelihood before starting reinforcement learning. However, such pre-training may have a negative effect in reinforcement learning. Consider the situation where the pre-training leads to zero cross-entropy loss. In this case, nothing will be learned during reinforcement learning because no exploratory action can be performed. Although pre-training in practice does not lead to zero cross-entropy loss, it can still overfit the data and result in very sharp out-2M 2M, SW Cross entropy 38.76 39.15 w/ beam search 39.88 40.35 REINFORCE w/ cross entropy 40.10 40.26 w/ beam search 40.36 40.38 w/ beam search ( K = 500) 40.07 40.07 w/ beam search ( K =2000) 40.30 40.50 w/ beam search ( K =3000) 40.27 40.41 Table 3: BLEU scores for the development split of the En-Ja (2M) and En-Ja (2M, SW) datasets. REINFORCE w/ cross entropy ( K =1000) 40.16 w/ beam search 40.50 Cross entropy (1.3M) w/ beam search 39.42 (Hashimoto and Tsuruoka, 2017) Cross entropy (2M) w/ beam search 40.29 (Oda et al., 2017b) Cross entropy (2M+1M back-trans.) 41.42 w/ beam search (Morishita et al., 2017) Table 4: BLEU scores for the En-Ja test split, where we use the En-Ja (2M, SW) dataset. put distributions, thereby hindering exploration in reinforcement learning. It is therefore important to consider a reinforcement learning setting with less or no pre-training (Liu et al., 2018). In Figure 3 for the En-Ja (100K) dataset, we show that the small softmax method works more effectively with fewer pre-training epochs. For this experiment, we set  = 0 in Equation (7) to purely focus on REINFORCE. Using GLEU (or BLEU) scores gives sparse rewards, and thus the resulting BLEU scores are very low with fewer pre-training steps, but the small softmax method has the potential to work well if we can design more effective reward functions. Results on larger datasets To see whether our method works in larger scales, Table 3 shows BLEU scores for the development split when using the En-Ja (2M) and En-Ja (2M, SW) datasets. 6 These results show that our method consistently works even on these larger datasets at the word and subword levels. In this table we also report how our method works with beam search, and the greedy-based BLEU scores are very close to those of beam search after the REINFORCE phase. When performing a beam search, we can optionally use different sizes of the small vocab-6 For the 2M dataset, the full softmax baseline achieves BLEU scores of 38.67 and 39.84 for the Cross entropy and REINFORCE w/ cross entropy settings, respectively. ulary, but we observe that our method is robust to the changes, whereas Wu et al. (2017) reported that their dynamic vocabulary selection method is sensitive to such changes. For reference, we report the test set results in Table 4. We cite BLEU scores from previously published papers which reported results of single models (i.e., without ensemble). Our method with greedy translation achieves a competitive score. It should be noted that Morishita et al. (2017) achieve a better score presumably because they used additional in-domain one million parallel sentences obtained by the back-translation technique (Sennrich et al., 2016). This section discusses our main contribution: how efficient our method is in accelerating reinforcement learning for sentence generation. We have examined the training-time efficiency of our method. Table 5 shows the training time [min-utes/epoch] for five different datasets. We selected the five datasets to show results with different vocabulary sizes and different maximum sentence lengths, and we observed the same trend on the other datasets. The vocabulary size | V | and the maximum sentence length max( N ) are shown for each training dataset. In the training with the standard cross entropy loss, the speedup by our method is not impressive as long as the vocabulary size | V | can be easily handled by the GPUs. We set S = 2 for the cross entropy training of the Full softmax method in the En-Ja (2M) setting, to reduce the GPU memory consumption as described in Section 4.4. In particular, in the En-Ja (2M) experiments, our method gains a factor of 2.7 speedup compared with the full softmax baseline ( S = 3 ). For most of the experimental settings, the speedup significantly accelerates our research and development cycles when working on reinforcement learning for sentence generation tasks. One exception is the Flickr8K dataset whose original vocabulary size | V | is already very small, and the lengths of the target sentences are short. In the supplementary material, we also show the test-time efficiency. Our method is also efficient in terms of GPU memory consumption at training time. Table 5 also shows the maximum GPU memory consumption during the training. These results show that our method easily fits in the memory of the single GTX 1080 GPU, whereas Full softmax is very sensitive to the vocabulary size | V | and the sentence lengths. In particular, we observe about 56% reduction in memory usage when using the En-Ja (2M) dataset. By saving the memory usage, one could try using larger models, larger mini-batches, larger vocabularies, and longer target sentences without relying on multiple GPUs. Scalability of our method To further show the memory efficiency our our method, we measured the GPU memory consumption with a larger mini-batch size, 2048. We applied the mini-batch splitting strategy to both the small and full softmax methods to handle such a large mini-batch size. In the En-Ja (2M) experiments with REINFORCE, our small softmax method works with the large batch-size by setting S = 6 , whereas the full softmax baseline needs S = 40 . Aggressively splitting the mini-batch (i.e. using larger values of S ) slows down the training time, and in that sense our method is much more efficient when we consider the larger mini-batch sizes. If we increase the mini-batch size to 4096, our small softmax method works with S = 12 . Reducing the computational cost at the large softmax layer in language modeling/generation is actively studied (Jean et al., 2015; Ji et al., 2016; Eriguchi et al., 2016; L'Hostis et al., 2016; Zoph et al., 2016; Wu et al., 2017). Most of the existing methods try to reduce the vocabulary size by either negative sampling or vocabulary prediction. One exception is that Oda et al. (2017a) propose to predict a binary code of its corresponding target word. Although such a sophisticated method is promising, we focused on the vocabulary reduction method to apply policy-based reinforcement learning in a straightforward way. As reported in this paper, one simple way to define a reward function for reinforcement learning is to use task-specific automatic evaluation metrics (Ranzato et al., 2016; Wu et al., 2016; Rennie et al., 2017; Zhang and Lapata, 2017; Paulus et al., 2018), but this is limited in that we can only use training data with gold target sentences. An alternative approach is to use a discriminator in generative adversarial networks (Goodfellow et al., 2014), and Yang et al. (2018) showed that REINFORCE with such a discriminator improves translation accuracy. However, Yang et al. (2018) only used the training data, and thus the potential of the generative adversarial networks is not fully realized. One promising direction is to improve the use of the generative adversarial networks for the sentence generation tasks by using our method, because our method can also accelerate the combination of REINFORCE and the discriminator. This paper has presented how to accelerate reinforcement learning for sentence generation tasks by reducing large action spaces. Our method is as accurate as, is faster than, and uses less GPU memory than the standard full softmax counterpart, on sentence generation tasks of different modalities. In future work, it is interesting to use our method in generative adversarial networks to further improve the sentence generation models. We thank anonymous reviewers for their fruitful comments. This work was supported by JST CREST Grant Number JPMJCR1513, Japan.
abstain abstain objective objective abstain result result abstain abstain abstain abstain abstain abstain method abstain objective abstain objective result abstain other abstain abstain method abstain other method abstain abstain method method abstain abstain method abstain abstain other abstain abstain method method abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain result objective result objective abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain method method method method abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain method method method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain result result abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain other other abstain abstain result abstain result result result abstain result result result result result abstain method method abstain method abstain method method method method abstain method method abstain method method abstain method method method method method other other other method method other other abstain method method result other other