Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pretraining task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).
Models such as BERT are pretrained language models which provide significant improvement for different tasks, however they suffer from high huge size and complexity. This paper has proposed using proximal gradient descent to find sparse weights for BERT to reduce the number of parameters and make the model smaller. They concentrate on the drawbacks of the previous sparse-based approaches and claimed that they have convergence issues (they have provided some evidence in the appendix). therefore, they propose to use reweighed sparse method and optimise it using proximal gradient descent which provides a closed form solution for sparse constraint. 
ALthough proposing a minor novelty (reweighted sparse optimization ), they have provided interesting results for both pretrained structure and fine-tuning for several different tasks. they have also provided some visualisation for the weight matrices after sparsification.
Their results are notably stronger than simply adding the L1 regularizer to the optimisation method. 
The paper is well written and easy to follow with nearly comprehensive related work.
However, there are some drawbacks:
1. They have claimed that “ To the best of our knowledge, we are the first to apply reweighted l1 and proximal algorithm in the DNN weight pruning domain, and achieve effective weight pruning on BERT. ”, however proximal optimization has been used for DNN in works like “Combined Group and Exclusive Sparsity for Deep Neural Networks, 2017”.  
2. It should be explained clearly about all the matrices included in the sparsification steps, despite only saying “parameters of the model”.
3. More analysis is required on the results, specially the diagrams for fine-tuning over different datasets.
4. It is essential to compare the method with other related works for Bert and transformer compression, including quantisation-based, factorisation-based, pruning, knowledge distillation papers such as:
--Prato, Gabriele, Ella Charlaix, and Mehdi Rezagholizadeh. "Fully Quantized Transformer for Improved Translation." arXiv preprint arXiv:1910.10485 (2019).
--Tang, Raphael, et al. "Distilling Task-Specific Knowledge from BERT into Simple Neural Networks." arXiv preprint arXiv:1903.12136 (2019).
--Sanh, Victor, et al. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv preprint arXiv:1910.01108 (2019).
--Ziheng Wang, et al. "Structured Pruning of Large Language Models."