Extreme Language Model Compression with Optimal Subwords and Shared Projections
Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks. However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices. In particular, the input word embedding matrix accounts for a significant proportion of the model*s memory footprint, due to the large input vocabulary and embedding dimensions. Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models. We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions. Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary. We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model. Our method is able to compress the BERT-BASE model by more than 60x, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7MB. Experimental results also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques.
This paper proposes techniques for improving the compression rate of Knowledge Distillation on BERT models. There are two main ideas. The first is the reduction of the parameter by reducing the vocabulary size of WordPiece model. The observation that a large fraction of parameters in BERT are dedicated to WordPiece embeddings is interesting, as this hasn*t received much attention in other works on BERT compression. The second idea is to add a loss function that measures the difference of parameters between teacher and student in the learned projection space. These two ideas are quite complementary in that one can apply one idea to vanilla knowledge distillation without having to apply another simultaneously. My major concern on this paper is that experiments are not designed to measure the impact of each design choice. First, it is unclear whether DualTrain-based methods outperform PKD with 3 layers because having more layers is more advantageous than having more hidden dimensions. While PKD doesn*t straightforwardly allow different hidden dimensions between teacher and student, it is a reasonably straightforward idea to train a shared projection matrix to enable it; concurrent work https://openreview.net/forum?id=rJx0Q6EFPB actually does that. If this was considered as too significant extension of PKD, then authors should*ve experimented with vanilla knowledge distillation with fewer hidden dimensions. Second, DualTrain-based models have both smaller vocabulary size and smaller hidden dimension size, and it is unclear which one played the greater role in successful distillation. I would suggest authors to carefully set up baselines that we shall precisely measure the impact of each design choice. Vanilla knowledge distillation with the same vocabulary as teacher and smaller hidden units shall allow us to understand the impact of decreasing hidden units. By comparing this with DualTrain models with the same vocabulary size, we shall now understand the benefit of decreasing the WordPiece vocabulary size. Authors make a very insightful note in the discussion that smaller WordPiece vocabulary would imply longer sequence lengths, which may degrade the performance of BERT models. This is an interesting observation, and authors should not avoid running experiments on tasks with longer sequences because it is likely to have negative results. By actually running experiments and breaking down metrics with different sequence length buckets, the research community shall better understand the impact of vocabulary size and the paper will be made stronger. Minor comments: 1) Table 3 is somewhat difficult to understand because each approach is making a different tradeoff between accuracy and model size. Plotting the accuracy metric vs. model size curve may more effectively visualize the tradeoff. 2) I don*t think the variable C in equation 3 was defined. Since there are two vocabularies, it might be useful to clarify the variable and clarify whether there is a loss function for words in teacher vocabulary.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Extreme Language Model Compression with Optimal Subwords and Shared Projections
Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks. However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices. In particular, the input word embedding matrix accounts for a significant proportion of the model*s memory footprint, due to the large input vocabulary and embedding dimensions. Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models. We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions. Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary. We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model. Our method is able to compress the BERT-BASE model by more than 60x, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7MB. Experimental results also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes techniques for improving the compression rate of Knowledge Distillation on BERT models. There are two main ideas. The first is the reduction of the parameter by reducing the vocabulary size of WordPiece model. The observation that a large fraction of parameters in BERT are dedicated to WordPiece embeddings is interesting, as this hasn*t received much attention in other works on BERT compression. The second idea is to add a loss function that measures the difference of parameters between teacher and student in the learned projection space. These two ideas are quite complementary in that one can apply one idea to vanilla knowledge distillation without having to apply another simultaneously. My major concern on this paper is that experiments are not designed to measure the impact of each design choice. First, it is unclear whether DualTrain-based methods outperform PKD with 3 layers because having more layers is more advantageous than having more hidden dimensions. While PKD doesn*t straightforwardly allow different hidden dimensions between teacher and student, it is a reasonably straightforward idea to train a shared projection matrix to enable it; concurrent work https://openreview.net/forum?id=rJx0Q6EFPB actually does that. If this was considered as too significant extension of PKD, then authors should*ve experimented with vanilla knowledge distillation with fewer hidden dimensions. Second, DualTrain-based models have both smaller vocabulary size and smaller hidden dimension size, and it is unclear which one played the greater role in successful distillation. I would suggest authors to carefully set up baselines that we shall precisely measure the impact of each design choice. Vanilla knowledge distillation with the same vocabulary as teacher and smaller hidden units shall allow us to understand the impact of decreasing hidden units. By comparing this with DualTrain models with the same vocabulary size, we shall now understand the benefit of decreasing the WordPiece vocabulary size. Authors make a very insightful note in the discussion that smaller WordPiece vocabulary would imply longer sequence lengths, which may degrade the performance of BERT models. This is an interesting observation, and authors should not avoid running experiments on tasks with longer sequences because it is likely to have negative results. By actually running experiments and breaking down metrics with different sequence length buckets, the research community shall better understand the impact of vocabulary size and the paper will be made stronger. Minor comments: 1) Table 3 is somewhat difficult to understand because each approach is making a different tradeoff between accuracy and model size. Plotting the accuracy metric vs. model size curve may more effectively visualize the tradeoff. 2) I don*t think the variable C in equation 3 was defined. Since there are two vocabularies, it might be useful to clarify the variable and clarify whether there is a loss function for words in teacher vocabulary.
assistant
This paper proposes techniques for improving the compression rate of Knowledge Distillation on BERT models. There are two main ideas. The first is the reduction of the parameter by reducing the vocabulary size of WordPiece model. The observation that a large fraction of parameters in BERT are dedicated to WordPiece embeddings is interesting, as this hasn*t received much attention in other works on BERT compression. The second idea is to add a loss function that measures the difference of parameters between teacher and student in the learned projection space. These two ideas are quite complementary in that one can apply one idea to vanilla knowledge distillation without having to apply another simultaneously. My major concern on this paper is that experiments are not designed to measure the impact of each design choice. First, it is unclear whether DualTrain-based methods outperform PKD with 3 layers because having more layers is more advantageous than having more hidden dimensions. While PKD doesn*t straightforwardly allow different hidden dimensions between teacher and student, it is a reasonably straightforward idea to train a shared projection matrix to enable it; concurrent work https://openreview.net/forum?id=rJx0Q6EFPB actually does that. If this was considered as too significant extension of PKD, then authors should*ve experimented with vanilla knowledge distillation with fewer hidden dimensions. Second, DualTrain-based models have both smaller vocabulary size and smaller hidden dimension size, and it is unclear which one played the greater role in successful distillation. I would suggest authors to carefully set up baselines that we shall precisely measure the impact of each design choice. Vanilla knowledge distillation with the same vocabulary as teacher and smaller hidden units shall allow us to understand the impact of decreasing hidden units. By comparing this with DualTrain models with the same vocabulary size, we shall now understand the benefit of decreasing the WordPiece vocabulary size. Authors make a very insightful note in the discussion that smaller WordPiece vocabulary would imply longer sequence lengths, which may degrade the performance of BERT models. This is an interesting observation, and authors should not avoid running experiments on tasks with longer sequences because it is likely to have negative results. By actually running experiments and breaking down metrics with different sequence length buckets, the research community shall better understand the impact of vocabulary size and the paper will be made stronger. Minor comments: 1) Table 3 is somewhat difficult to understand because each approach is making a different tradeoff between accuracy and model size. Plotting the accuracy metric vs. model size curve may more effectively visualize the tradeoff. 2) I don*t think the variable C in equation 3 was defined. Since there are two vocabularies, it might be useful to clarify the variable and clarify whether there is a loss function for words in teacher vocabulary.