Universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. A common paradigm is to pre-train a feature extractor on large amounts of data then fine-tune it as part of a deep learning model on some downstream task (i.e. transfer learning). While effective, feature extractors like BERT may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.
This work is an empirical study of testing how pruning at the pre-training stage affects subsequent transfer learning (through fine-tuning) stage. The main idea is to carefully control the amount of sparsity injected into BERT through weight magnitude pruning and study the impact on accuracy. The experimental setup is mostly well done, especially the part that disentangles the complexity restriction and information deletion. During the exploration, the authors made several interesting observations, such as 30-40% model weights do not encode any useful inductive bias, which could help shed some light for future work on both training and compressing BERT-like models.
Overall, the paper is well written and explained. The goal is meaningful, and this is a sensible contribution to the ongoing interests of compressing BERT-like large models for efficient training and inference. 
My major concern is on its novelty and how directly it can provide benefit to computation.  First, although the findings are interesting, the methods used in this paper are not new.  Various pruning techniques have been explored in prior work, which makes the novelty contribution of this paper somewhat limited. 
Furthermore, the study has mostly focused on the impact of random sparsity to accuracy. However, as it is known that it is really difficult for modern hardware to benefit from random sparsity because it leads to irregular memory accesses, which negatively impact the performance. It has been observed that speedups are very limited or can be negative even the random sparsity is >95% [1]. Therefore, it is hard to judge how inference or training can benefit from 30-40% weight sparsity. Going forward, the authors are encouraged to choose pruning methods that lead to regular memory access to avoid adversely impacting practical acceleration in modern hardware platforms.
[1] Learning Structured Sparsity in Deep Neural Networks. Wen et al. NeurIPS 2016