LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need 2 |V | vectors to represent a vocabulary of |V | unique words, which are far less than the |V | vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm LightRNN to reflect its very small model size and very high training speed.
Consider a word level language model. The majority of its parameters lie in the embedding matrix, as each word is typically assigned a 100-1000 dimensional vector. This submission presents a novel factorization of the embedding matrix, that quadratically reduces the memory requiremnts. In turn, this allows the use of larger word embeddings and leads to state-of-the-art performance on ACLW and Billion Words benchmarks.  Traditionally, word embeddings are kept in a single table of size #words x embedding_dim. In this contribution, the authors create two small tables, both of size sqrt(#words) x embedding-dim, that are called the row-embedding matrix and the column-embedding matrix. Each word is assigned to an embedding from the row-embedding matrix and from the column-embedding one. Thus the embeddings of words are shared between words. However, this change substantially reduces the memory requirements.  During model use, the RNN makes two steps for each word - it updates its hidden state first using the row-embedding matrix, then using the column-embedding matrix. Likewise, the probability of sampling a word is factorized into two probabilities: of the probability assigned to the word's position of the word in the row-matrix, and of of the probability assigned to the word's position in the column matrix. This solves another important problem, by reducing the amount of computation needed to compute the probability of an output word.  The key ingredient of the proposed contribution is an algorithm that assigns word and column indexes to words, therefore establishing the sharing pattern for the embeddings. The authors frame it as a matching problem, and elegantly use a heuristic solution to the Minimum Cost Maximum Flow graph problem.   
This paper solves two important practical problems of 1. reducing the size of the word embedding matrix and 2. reducing the computations of word probabilities, obviating the need for a hierarchical Softmax or Monte Carlo estimations of the model's training cost. The proposed model makes no assumptions about the structure of the words, which makes it potentially useful outside of NLP. In contrast, character-based word embedding models also reduce the model size, but do need to access the internal structure of words (i.e. their characters).  The presentation is sufficiently clear, though I have the following questions: 1. from equation (1) and Fig. 2 it seems that the model sees the t-th word before predicting it. Still, there is an information bottleneck, as the word is aliased with sqrt(#words) that are assigned the same row-embedding. However, this is still surprising - could the author comment whether this is intentional, and how do the models behave when the word w_t is generated using information about words up to t-1 only?  2. how is the cost (4) computed? I guess that there are separate matrices for input-word embedding and  output-word embedding, that share the word-allocation table. To compute eq. (4) is only the output allocation changed (which is fast to compute, as the hidden states of the RNN do not change and words are conditionally independent given the hidden states), or is the input allocation changed too?   3. how often in practice is the index reallocation (step 3) phase run?