Recurrent neural networks have achieved state-of-the-art results in many artificial intelligence tasks, such as language modeling, neural machine translation, speech recognition and so on. One of the key factors to these successes is big models. However, training such big models usually takes days or even weeks of time even if using tens of GPU cards. In this paper, we propose an efficient architecture to improve the efficiency of such RNN model training, which adopts the group strategy for recurrent layers, while exploiting the representation rearrangement strategy between layers as well as time steps. To demonstrate the advantages of our models, we conduct experiments on several datasets and tasks. The results show that our architecture achieves comparable or better accuracy comparing with baselines, with a much smaller number of parameters and at a much lower computational cost. Recurrent Neural Networks (RNNs) have been widely used for sequence learning, and achieved state-of-the-art results in many artificial intelligence tasks in recent years, including language modeling (Zaremba et al., 2014; Merity et al., 2017), neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), and speech recognition (Graves et al., 2013). To get better accuracy, recent state-of-the-art RNN models are designed toward big scale, include going deep (stacking multiple recurrent layers) (Pascanu et al., 2013a) and/or going wide (in-creasing dimensions of hidden states). For example, an RNN based commercial Neural Machine Translation (NMT) system would employ tens of layers in total, resulting in a large model with hundreds of millions of parameters (Wu et al., 2016). However, when the model size increases, the computational cost, as well as the memory needed for the training, increases dramatically. The training cost of aforementioned NMT model reaches as high as 10 19 FLOPs, and the training procedure spends several days with even 96 GPU cards (Wu et al., 2016)  such complexity is prohibitively expensive. While above models benefit from big neural networks, it is observed that such networks often have redundancy of parameters (Kim and Rush, 2016), motivating us to improve parameter efficiency and design more compact architectures that are more efficient in training while keeping good performance. Recently, many efficient architectures for convolution neural networks (CNNs) have been proposed to reduce training cost in computer vision domain. Among them, the group convolution is one of the most widely used and successful attempts (Szegedy et al., 2015; Chollet, 2016; Zhang et al., 2017b), which splits the channels into groups and conducts convolution separately for each group. It's essentially a diagonal sparse operation to the convolutional layer, which reduces the number of parameters as well as the computation complexity linearly w.r.t. the group size. Empirical results for such group convolution optimization show great speed up with small degradation on accuracy. In contrast, there are very limited attempts for designing better architectures for RNNs. Inspired by those works on CNNs, in this paper, we generalize the group idea to RNNs to conduct recurrent learning in the group level. Different from CNNs, there are two kinds of parameter redundancy in RNNs: (1) the weight matrices transforming a low-level feature representation to a high-level one may contain redundancy, and (2) the recurrent weight matrices transferring the hidden state of the current step to the hidden state of the next step may also contain redundancy. Therefore, when designing efficient RNNs, we need to consider both the kinds of redundancy. We present a simple architecture for efficient sequence learning which consists of group recurrent layers and representation rearrangement layers. First, in a recurrent layer, we split both the input of the sequence and the hidden states into disjoint groups, and do recurrent learning separately for each group. This operation clearly reduces the model complexity, and can learn intragroup features efficiently. However, it fails to capture dependency cross different groups. To recover the inter-groups correlation, we further introduce a representation rearrangement layer between any two consecutive recurrent layers, as well as any two time steps. With these two operations, we explicitly factorize a recurrent temporal learning into intra-group temporal learning and inter-group temporal learning with a much smaller number of parameters. The group recurrent layer we proposed is equivalent to the standard recurrent layer with a block-diagonal sparse weight matrix. That is, our model employs a uniform sparse structure which can be computed very efficiently. To show the advantages of our model, we analyze the computation cost and memory usage comparing with standard recurrent networks. The efficiency improvement is linear to the number of groups. We conduct experiments on language modeling, neural machine translation and abstractive summarization by using a state-of-the-art RNN architecture as baseline. The results show that our model can achieve comparable or better accuracy, with a much smaller number of parameters and in a shorter training time. The remainder of this paper is organized as follows. We first present our newly proposed architecture and conduct in depth analysis on its efficiency improvement. Then we show a series of empirical study to verify the effectiveness of our methods. Finally, to better position our work, we introduce some related work and then conclude our work. In this section, we introduce our proposed architecture for RNNs. Before getting into the details of the group recurrent layer and representation rearrangement layer in our architecture, we first revisit the vanilla RNNs. An RNN is a neural network with recurrent layers that capture temporal dynamics of a sequence with arbitrary length. It recursively applies a transition function to its internal hidden state for each symbol of input sequence. The hidden state at time step t is computed as a function f of the current input symbol x t and the previous hidden state h t  1 in a recurrent form: h t = f ( x t , h t  1 ) . where W is the input-to-hidden weight matrix, U is the state-to-state recurrent weight matrix, and tanh is the hyperbolic tangent function. Our work is independent to the choices of the recurrent function ( f in Equation 1). For simplicity, in the following, we take the vanilla RNN as an example to introduce and analyze our new architecture. We aim to design an efficient RNN architecture by reducing the parameter redundancy while keeping accuracy at the same time. Inspired by the success of group convolution in CNN, our architecture employs the group strategy to achieve a sparsely connected structure between neurons of recurrent layers, and employs the representation rearrangement to recover the correlation that may destroyed by the sparsity. At a high level, we explicitly factorize the recurrent learning as intergroup recurrent learning and intra-group recurrent learning. In the following, we will describe our RNN architecture in detail, which consists of a group recurrent layer for intra-group correlation and a representation rearrangement layer for intergroup correlation. For standard recurrent layer, the model complexity increases quadratically with the dimension of hidden state. Suppose the input x is with dimension M , while the hidden state is with dimension N . Then, for standard vanilla RNN cell, according to Equation 2, the number of parameters, as well as the computation cost is N 2 + N  M. (3) It's obvious that the hidden state dimension largely determines the model complexity. Optimization on reducing computation w.r.t the hidden state is the key to improve the overall efficiency. Accordingly, we present a group recurrent 800 layer which adopts a group strategy to approximate the standard recurrent layer. Specifically, we consider to split both the input x t and hidden state h t into K disjoint groups as { x 1 t , x 2 t , ..., x K t } and { h 1 t , h 2 t , ..., h Kt } respectively, where x it , h it represent the input and hidden state for i -th group at time step t . Based on this split, we then perform recurrent computation in every group independently. This will captures the intra-group temporal correlation within the sequence. Formally, in the group recurrent layer, we first compute the hidden state of each group h it as h it = f i ( x it , h it  1 ) , i = 1 , 2 , ..., K. (4) Then, concatenating all the hidden states from each group together, h t = concat ( h 1 t , h 2 t , ..., h Kt ) (5) we get the output of the group recurrent layer. The group recurrent layer is illustrated as Figure 2(a) and Figure 1(b). Obviously, by splitting the features and hidden states into K groups, the number of parameters and the computation cost of recurrent layer reduce to K  (( N K ) 2 + N K  M K ) = N 2 + N  M K (6) Comparing Equation 3 with Equation 6, the group recurrent is K times more efficient than the standard recurrent layer, in terms of both computational cost and number of parameters. Although the theoretical computational cost is attractive, the speedup ratio also depends on the implementation details. A naive implementation of Equation 4 would introduce a for loop, which is not efficient since the additional overhead and poor parallelism. In order to really achieve linear speed up, we employ a batch matrix multiplication to assemble the computation of different groups in a single round of matrix multiplication. This operation is critical especially when each group isn't big enough to fully utilize the entire GPU computation power. Group recurrent layer is K times more efficient comparing with the standard recurrent layer. But, it only captures the temporal correlation inside a single feature group and fails to learn dependency across features from different groups. more specifically, the internal state of RNN only contains history from corresponding group (Figure 801 1(b)). Similar problem also exists in the vertical direction of group recurrent layers (Figure 2(a)). Consider a network with multiple stacked group recurrent layers, the output of the specific group are only get from the corresponding input group. Obviously, there will be a significant drop of representation power since many feature correlations are cut off by this architecture. To recover the inter-group correlations, one simple way is adding a projection layer to transform the hidden state outputted by the group recurrent layer, like the 1  1 convolution used in depthwise separable convolutional (Chollet, 2016). However, such method would bring additional N 2 computation complexity and model parameters. Inspired by the idea of permuting channels between convolutional layers in recent CNN architectures (Zhang et al., 2017a,b), we propose to add representation rearrangement layer between consecutive group recurrent layers (Figure 1(c)), as well as the time steps within a group recurrent layer (Figure 2(b)). The representation rearrangement aims to rearrange the hidden representation, to make sure the subsequent layers, or time steps, can see features from all input groups. The representation rearrangement layer is parameter-free and simple. We leverage the same implementation in (Zhang et al., 2017b) to conduct the rearrangement. It's finished with basic tensor operations reshape , and transpose , which brings (almost) no runtime overhead in our experiments. Consider the immediate representation h t  RN outputted by group recurrent layer with group number K . First, we reshape the representation to add an additional group dimension, resulting in a tensor with new shape ( K, N/K ) . Second, we transpose the two dimensions of the temporary tensor, changing the tensor shape to ( N/K, K ) . Finally, we reshape the tensor along the first axis to restore the representation to its original shape (a vector of size N ). Figure 3 illustrates the operations with a simple example whose representation is with size 8 and group number is 2. Combining the group recurrent layer and representation rearrangement layer, we rebuild the recurrent layer into an efficient and effective layer. We note that, different from convolutional neural networks that are only deep in space, the stack RNNs are deep in both space and time. Figure 1 illustrates our architecture along the spatial direction, and Figure 2 illustrates our architecture along the temporal direction. By applying group operation and representation rearrangement in both space and time, we build a new recurrent neural network with high efficiency. In this section, we analyze the relation between group recurrent layer and standard recurrent layer, and discuss the advantages of group recurrent networks. From the reformulation, we can see group recurrent layer is equivalent to standard recurrent layer with block-diagonal sparse weight matrix. Our method employs a group level sparsity in recurrent computation, leading to a uniform sparse structure. This uniform sparse structure can enjoy the efficient computing of dense matrix, as we discussed in Section 2.1. This reformulation also shows that there is no connection across neurons in different groups. Increasing the group number will lead to higher sparse rate. This sparse structure may limit the representation ability of our model. In order to recover the correlation across different groups, we add representation rearrangement to make up for representation ability. We have shown that with same width of recurrent layer, our architecture with group number K achieves a compact model, which has K times less number of parameters than the standard recurrent network. Therefore with same number of parameters, group recurrent networks can provide more possibility to try more complex model without any 802 Figure 3: Illustration of the implementation of representation rearrangement with basic tensor operation. additional computation and parameter overhead. Given a standard recurrent neural network, we can construct a corresponding group recurrent neural network with same number of parameters, but with K times wider, or with K times deeper. A factor smaller than K would make our networks still effective than standard recurrent network, but with wider and/or deeper recurrent layers. This could somehow compensate the potential performance drop due to the aggressive sparsity when group number is too large. Therefore, our architecture provides large model space to find a better tradeoff between parameter and performance given a fixed resource budget. And our model is a more effective RNN architecture when the network goes deeper and wider. At last, we note that our architecture focuses on improving the efficiency of recurrent layers. Thus the whole parameter and computational cost reduction depend on the ratio of recurrent layer in the entire network. Consider a text classification task, a often used RNN model would introduce an embedding layer for the input tokens and a softmax layer for the output, making the parameter reduction and speedup for the whole network is not strictly linear with the group number. However, we argue that for deeper and/or wider RNN whose recurrent layers dominate the parameter and computational cost, our method would enjoy more efficiency improvement. In this section, we present results on three sequence learning tasks to show the effectiveness of our method: 1). language modeling; 2). neural machine translation; 3). abstractive summarization. For evaluating the effectiveness of our approach, we perform language modeling over Penn Tree-bak (PTB) dataset (Marcus et al., 1993). We use the data preprocessed by (Mikolov et al., 2010) 1 , which consists of 929 K training words, 73 K validation words, and 82 K test words. It has 10 K words in its vocabulary. We compare our method (named Group LSTM) with the standard LSTM baseline (Zaremba et al., 2014) and its two variants with Bayesian dropout (named LSTM + BD) (Gal and Ghahramani, 2016) and with word tying (named LSTM + WT) (Press and Wolf, 2017). Following the big model settings in (Zaremba et al., 2014; Gal and Ghahramani, 2016; Inan et al., 2016) , all experiments use a two-layer LSTM with 1 , 500 hidden units and an embedding of size 1 , 500 . We set group number 2 in this experiment since PTB is a relative simple dataset. We use Stochastic Gradient Descent (SGD) to train all models. Results We compare the word level perplexity obtained by the standard LSTM baseline models and our group variants, in which we replace the standard LSTM layer with our group LSTM layer. As shown in Table 1, Group LSTM achieves comparable performance with the standard LSTM baseline, but with a 27 % parameter reduction. A variant using Bayesian dropout (BD) is proposed by (Gal and Ghahramani, 2016) to prevent over-fitting and improve performance. We test our model with LSTM + BD, achieving similar results with above comparison. Finally, we compare our model with the recently proposed word tying (WT) technology, which ties input embedding and output embedding with same weights. Our model achieves even better perplexity than the results reported by (Press and Wolf, 2017). Since word tying reduces the number of parameters of embedding and softmax layers, thus improving the ratio of LSTM layer parameter. Our method achieves a 35 % parameter reduction. We then study our model in neural machine translation. We conduct experiments on two translation tasks, German-English task (De-En for short) and English-German task (En-De for short). For De-En translation, we use data from the De-En ma-1 http://www.fit.vutbr.cz/imikolov/ rnnlm/simple-examples.tgz 803 Model Parameters Validation Set Test Set LSTM (Zaremba et al., 2014) 66M 82.2 78.4 2 Group LSTM 48M 82.0 78.6 LSTM + BD (Gal and Ghahramani, 2016) 66M 77.9 75.2 2 Group LSTM + BD 48M 79.9 75.8 LSTM + WT (Press and Wolf, 2017) 51M 77.4 74.3 2 Group LSTM + WT 33M 76.8 73.3 LSTM + BD + WT (Press and Wolf, 2017) 51M 75.8 73.2 2 Group LSTM + BD + WT 33M 75.6 71.8 Table 1: Single model complexity on validation and test sets for the Penn Treebank language modeling task. chine translation track of the IWSLT 2014 evaluation campaign (Cettolo et al., 2014). We follow the pre-processing described in previous works (Wu et al., 2017). The training data comprises about 153 K sentence pairs. The size of validation data set is 6 , 969 , and the test set is 6 , 750 . For En-De translation, we use a widely adopted dataset (Jean et al., 2015; Wu et al., 2016). Specifically, part of data in WMT'14 is used as the training data, which consists of 4.5M sentences pairs. newstest2012 and newstest2013 are concatenated as the validation set and newstest2014 acts as test set. These two datasets are preprocessed by byte pair encoding (BPE) with vocabulary of 25 K and 30 K for De-En and En-De respectively, and the max length of sub-word sentence is 64 . Our model is based on RNNSearch model (Bah-danau et al., 2014), but replacing the standard LSTM layer with our group LSTM layer. Therefore, we name our model as Group RNNSearch model. The model is constructed by LSTM encoder and decoder with attention, where the first layer of encoder is bidirectional LSTM. For De-En, we use two layers for both encoder and decoder. The embedding size is 256 , which is same as the hidden size for all LSTM layers. As for EnDe, we use four layers for encoder and decoder 2 . The embedding size is 512 and the hidden size is 1024 3 . All the models are trained by Adadelta (Zeiler, 2012) with initial learning rate 1 . 0 . The gradient is clipped with threshold 2 . 5 . The mini-batch size is 32 for De-En and 128 for En-De. We use dropout (Srivastava et al., 2014) with rate 0 . 1 for all layers except the layer before softmax with 0 . 5 . We halve the learning rate according to the validation performance. 2 For easy to implement, we still keep the first layer with attention computation in the decoder as original LSTM layer. 3 In our implementation, suppose the hidden size is d , after the first bi-directional LSTM layer in the encoder, the hidden size of the above LSTM layers in the encoder should be 2  d . Results We compute tokenized case-sensitive BLEU (Papineni et al., 2002) 4 score as evaluation metric. For decoding, we use beam search (Sutskever et al., 2014) with beam size 5 . From Table 2, we can observe that on De-En task, Group RNNSearch models achieve comparable or better BLEU score compared with the RNNSearch but with much less number of parameters. Specifically, with group number 2 and 4, we achieve about 28% and 43% parameter reduction of recurrent layers respectively. Note that our results also outperform the state-of-the-art result reported in NPMT (Huang et al., 2017). The En-De translation results are shown in Table 3. We compare our Group RNNSearch models with Google's GNMT system (Wu et al., 2016) and DeepLAU (Wang et al., 2017). Our 4 4 https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/generic/multi-bleu.perl 804 Group RNNSearch model achieves 23 . 61 , which is comparable to DeepLAU ( 23 . 80 ). Our 2 Group RNNSearch model achieves a BLEU score of 23 . 93 , slightly less than GNMT ( 24 . 61 ), but outperforms the DeepLAU. More importantly, our Group RNNSearch models decrease more than 30% and 50% RNN parameters with 2 groups and 4 groups respectively compared with GNMT. At last, we valid our approach on abstractive summarization task. We train on the Gigaword corpus (Graff and Cieri, 2003) and pre-process it identically to (Rush et al., 2015; Shen et al., 2016), resulting in 3 . 8 M training article-headline pairs, 190 K for validation and 2 , 000 for test. Similar to (Shen et al., 2016), we use a source and target vocabulary consisting of 30 K words. The model is almost same as the one used in De-En machine translation, which is a two layers RNNSearch model, except that the embedding size is 512 , and the LSTM hidden size in both encoder and decoder is 512 . The initial values of all weight parameters are uniformly sampled between (  0 . 05 , 0 . 05) . We train our Group RNNSearch model by Adadelta (Zeiler, 2012) with learning rate 1 . 0 and gradient clipping threshold 1 . 5 (Pas-canu et al., 2013b). The mini-batch size is 64 . Results We evaluate the summarization task by commonly used ROUGE (Lin, 2004) F1 score. During decoding, we use beam search with beam size 10. The results are shown in Table 4. From Table 4, we can observe that the performance is consistent with machine translation task. Our Group RNNSearch model achieves comparable results with RNNSearch, and our 2 Group RNNSearch model even outperforms RNNSearch baseline. Besides, we compare with several other widely adopted methods, our models also show strong performance. Therefore, we can keep the good performance even though we reduce the parameters of the recurrent layers by nearly 50% , which greatly proves the effectiveness of our method. In addition to showing that group RNN can achieve competing or better performance with much less number of parameters, we further study the effect of group number to training speed and convergence, and the effect of representation rear-Model rangement to performance. Due to space limitation, we only report results for language modeling on PTB dataset; for other tasks we have similar results. In Figure 4, the left one shows that how number of parameters and training speed vary when group number ranging from 1 to 16. We can see that the number of parameters (of recurrent layers) is reduced linearly when increasing number of groups. In the meantime, we also achieves substantial speed up about throughput when increasing group number. We note that the speedup is sub-linear instead of linear since our method focuses on the speedup on recurrent layers, as discussed in Section 3.2. Besides, we also compare the convergence curve in the right of Figure 4, which shows that our method (almost) doesn't slow down the convergence in terms of epoch number. Considering the throughput speedup of our method, we can accelerate training by a large margin. At last, we study the role that representation rearrangement layer plays in our architecture. We compare Group LSTM with and without representation rearrangement between layers and time steps, with the group number 2 and 4 respectively. From Table 5, we can see that the models with representation rearrangement consistently outperforms the ones without representation rearrangement. This shows the representation rearrangement is critical for group RNN. Improving RNN efficiency for sequence learning is a hot topic in recent deep learning research. For parameter and computation reduction, LightRNN (Li et al., 2016) is proposed to solve big vocabulary problem with a 2-component shared embedding, while our work addresses the parameter redundancy caused by recurrent layers. To speed up RNN, Persistent RNN (Diamos et al., 2016) is proposed to improve the RNN computation throughput by mapping deep RNN efficiently onto GPUs, which exploits GPU's inverted memory hierarchy to reuse network weights over multiple time steps. (Neil et al., 2017) proposes delta networks for optimizing the matrix-vector multiplications in RNN computation by considering the temporal properties of the data. Quasi-RNN (Bradbury et al., 2016) and SRU (Lei and Zhang, 2017) are proposed for speeding up RNN computation by designing novel recurrent units which relax dependency between time steps. Different from these works, we optimize RNN from the perspective of network architecture innovation by adopting a group strategy. There is a long history about the group idea in deep learning, especially in convolutional neural networks, aiming to improve the computation efficiency and parameter efficiency. Such works can date back at least to AlexNet (Krizhevsky et al., 2012), which splits the convolutional layers into 2 independent groups for the ease of model-parallelism. The Inception (Szegedy et al., 2015) architecture proposes a module that employs uniform sparsity to improve the parameter efficiency. Going to the extreme of Inception, the Xception (Chollet, 2016) adopts a depthwise separable convolution, where each spatial convolution only works on a single channel. MobileNet (Howard et al., 2017) uses the same idea for efficient mobile model. IGCNet (Zhang et al., 2017a) and ShuffleNet (Zhang et al., 2017b) also adopt the group convolution idea, and further permute the features across consecutive layers. Similar to these works, we also exploit the group strategy. But we focus on efficient sequence learning with RNN, which, different from CNN, contains an internal memory and an additional temporal direction. In the RNN literature, there is only one paper (Kuchaiev and Ginsburg, 2017), to our best knowledge, exploiting the group strategy. However, this work assumes the features are group independent, thus failing to capturing the inter-group correlation. Our work employs a representational rearrangement mechanism, which avoids the assump-tion and improves the performance, as shown in our empirical experiments. We have presented an efficient RNN architecture for sequence learning. Our architecture employs a group recurrent layer to learn intra-group correlation efficiently, and representation rearrangement layer to recover inter-group correlation for keeping representation ability. We demonstrate our model is more efficient in terms of parameters and computational cost. We conduct extensive experiments on language modeling, neural machine translations and abstractive summarization, showing that our method achieves competing performance with much less computing resource.
abstain abstain abstain objective objective result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method objective abstain abstain abstain abstain objective method result abstain method result method objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method result method abstain abstain abstain method method result abstain abstain method abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result result abstain abstain other abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result result abstain result abstain result abstain result result abstain result abstain abstain abstain result abstain other abstain other other other method other other other other other other abstain method abstain method abstain method method objective result